notmuch/notmuch-new.c, branch 0.6

Remove some variables which were set but not used.

2011-05-11T20:27:14Z

gcc (at least as of version 4.6.0) is kind enough to point these out to us, (when given -Wunused-but-set-variable explicitly or implicitly via -Wunused or -Wall). One of these cases was a legitimately unused variable. Two were simply variables (named ignored) we were assigning only to squelch a warning about unused function return values. I don't seem to be getting those warnings even without setting the ignored variable. And the gcc docs. say that the correct way to squelch that warning is with a cast to (void) anyway.

new: Update comments for add_files_recursive

2011-03-10T19:56:16Z

The most recent commit optimized the implementation of this function. This commit simply updates the relevant comments to match the new implementation.

new: read db_files and db_subdirs only if mtime changed

2011-03-10T19:48:33Z

The db_files and db_subdirs are unnecessary for unchanged directories. maildir with 10000 e-mails: old version: $ time ./notmuch new No new mail. real 0m0.053s user 0m0.028s sys 0m0.026s new version: $ time ./notmuch new No new mail. real 0m0.032s user 0m0.009s sys 0m0.023s Signed-off-by: Karel Zak Reviewed-by: Austin Clements Looks good (faster than, but provably equivalent to the original code! notmuch_directory_get_child_* are side-effect free, db_files/db_subdirs aren't used between where they were set in the old code and where they are set in the new code, and db_files/db_subdirs are initialized to NULL when declared). Another timing data point: Old code: ./notmuch new 0.77s user 0.28s system 99% cpu 1.051 total New code: ./notmuch new 0.09s user 0.27s system 98% cpu 0.368 total

new: Print progress estimates only when we have sufficient information

2011-01-26T13:47:51Z

Without this patch, it might happen that the remaining time or processing rate were calculated just after start where nothing was processed yet. This resulted into division by a very small number (or zero) and the printed information was of little value. Instead of printing nonsenses we print only that the operation is in progress. The estimates will be printed later, after there is enough data.

new: Enhance progress reporting

2011-01-26T12:10:11Z

notmuch new reports progress only during the "first" phase when the files on disk are traversed and indexed. After this phase, other operations like rename detection and maildir flags synchronization are performed, but the user is not informed about them. Since these operations can take significant time, we want to inform the user about them. This patch enhances the progress reporting facility that was already present. The timer that triggers reporting is not stopped after the first phase but continues to run until all operations are finished. The rename detection and maildir flag synchronization are enhanced to report their progress.

new: Add all initial tags at once

2011-01-26T12:05:28Z

If there are several tags applied to the new messages, it is beneficial to store them to the database at one, because it saves some time, especially when the notmuch new is run for the first time. This patch decreased the time for initial import from 1h 35m to 1h 14m.

Do not defer maildir flag synchronization for new messages

2011-01-26T11:52:54Z

This is a simplified version of a patch originally by Michal Sojka which is designed to have the same performance benefits. Michal said the following: When notmuch new is run for the first time, it is not necessary to defer maildir flags synchronization to later because we already know that no files will be removed. Performing the maildinr flag synchronization immediately after the message is added to the database has the advantage that the message is likely hot in the disk cache so the synchronization is faster. Additionally, we also save one database query for each message, which must be performed when the operation is deferred. Without this patch, the first notmuch new of 200k messages (3 GB) took 1h and 46m out of which 20m was maildir flags synchronization. With this patch, the whole operation took only 1h and 36m. Unlike Michal's patch, this version does the deferral for any new message, rather than doing it only on the first run of "notmuch new".

notmuch new: Scan directory whenever fs mtime is not equal to db mtime

2010-12-05T09:40:16Z

Previously, we would only scan a directory if the filesystem modification time was strictly newer than the database modification time for the directory. This would cause a problem for systems with an unstable clock, (if a new mail was added to the filesystem, then the system clock rolled backward, "notmuch new" would not find the message until the clock caught up and the directory was modified again). Now, we always scan the directory if the modification time of the directory is not exactly the same between the filesystem and the database. This avoids the problem described above even with an unstable system clock.

notmuch new: Defer maildir_flags synchronization until after removals

2010-11-11T11:40:19Z

When a file in the mailstore is renamed, this appears to "notmuch new" as both an added file and a removed file (for the same message). We want the synchronization of the maildir_flags to reflect the final state, (after the rename is complete). Therefore, it's incorrect to perform the synchronization immediately after adding a new file. Instead we queue up these synchronizations (by message ID[*]) and perform them after the removals are complete. With this change, the "dump/restore" case of the maildir-sync tests, as well as the recent "remove 'S'" case both now pass where they were failing before. Interestingly, the "remove info" test was passing before, but now fails. This is actually due to a separate bug, (and the bug just fixed was masking it, by preventing the test from performing as desired). [*] It's important to queue by message ID---queueing actual message objects does not work since the message objects will retain stale data such as the old filenames.

lib: Rework interface for maildir_flags synchronization

2010-11-11T11:40:19Z

Instead of having an API for setting a library-wide flag for synchronization (notmuch_database_set_maildir_sync) we instead implement maildir synchronization with two new library functions: notmuch_message_maildir_flags_to_tags and notmuch_message_tags_to_maildir_flags These functions are nicely documented here, (though the implementation does not quite match the documentation yet---as plainly evidenced by the current results of the test suite).