+Performance Tests
+-----------------
+
+This directory contains two kinds of performance tests: time tests,
+and memory tests. The former use gnu time, and the latter use
+valgrind.
+
Pre-requisites
--------------
- gpg
- gnu tar
-- gnu time
+- gnu time (for the time tests)
- xz. Some speedup can be gotten by installing "pixz", but this is
probably only worthwhile if you are debugging the tests.
+- valgrind (for the memory tests)
+- perf (optional, for more fine-grained timing)
Getting set up to run tests:
----------------------------
First, you need to get the corpus. If you don't already have the gpg
key for David Bremner, run
- % gpg --search 'david@tethera.net'
+ % gpg --locate-external-key 'david@tethera.net'
This should get you a key with fingerprint
- 815B 6398 2A79 F8E7 C727 86C4 762B 57BB 7842 06AD
+ 7A18 807F 100A 4570 C596 8420 7E4E 65C8 720B 706B
(the last 8 digits are printed as the "key id").
In case that fails or is too slow, check
- http://notmuchmail.org/corpus
+ https://notmuchmail.org/corpus
for a list of mirrors.
Running tests
-------------
-The easiest way to run performance tests is to say "make perf-test", (or
-simply run the notmuch-perf-test script). Either command will run all
-available performance tests.
-
-Alternately, you can run a specific subset of tests by simply invoking
-one of the executable scripts in this directory, (such as ./basic).
-Each test script supports the following arguments
+The easiest way to run performance tests is to say "make perf-test".
+This will run all time and memory tests. Be aware that the memory
+tests are quite time consuming when run on the full corpus, and that
+depending on your interests it may be more sensible to run "make
+time-test" or "make memory-test". You can also invoke one of the
+scripts notmuch-time-test or notmuch-memory-test or run a more
+specific subset of tests by simply invoking one of the executable
+scripts in this directory, (such as ./T00-new). Each test script
+supports the following arguments
--small / --medium / --large Choose corpus size.
--debug Enable debugging. In particular don't delete
- temporary directories.
+ temporary directories.
+--perf Run perf record in place of /usr/bin/time. Perf output can be
+ found in a log directory.
+--call-graph {fp,lbr,dwarf} Call graph option for perf record. Default is 'lbr'.
+
+When using the make targets, you can pass arguments to all test
+scripts by defining the make variable OPTIONS.
+
+Log Directory
+-------------
+
+The memory tests, and the time tests when option '--perf' is given
+save their output in a directory named as follows
+
+ log.$test_name-$corpus_size-$timestamp
+
+These directories are removed by "make clean".
Writing tests
-------------
-Have a look at "basic" for an example. Sourcing "perf-test-lib.sh" is
-mandatory. Utility functions include
+Have a look at "T01-dump-restore" for an example time test and
+"M00-new" for an example memory test. In both cases sourcing
+"perf-test-lib.sh" is mandatory.
-- 'add_email_corpus' unpacks a set of messages and adds them to the database.
-- 'cache_database': makes a snapshot of the current database
-- 'uncache_database': forces the next 'add_email_corpus' to rebuild the
- database.
-- 'time_start' unpacks the mail corpus and calls notmuch new if it
+Basics:
+
+- '(time|memory)_start' unpacks the mail corpus and calls notmuch new if it
cannot find a cache of the appropriate corpus.
-- 'time_done' does the cleanup; comment it out or pass --debug to the
+- '(time|memory)_run' runs the command under time or valgrind. Currently
+ "memory_run" does not support i/o redirection in the command.
+- '(time|memory)_done' does the cleanup; comment it out or pass --debug to the
script to leave the temporary files around.
+
+Utility functions include
+
+- 'add_email_corpus' unpacks a set of messages and tags
+- 'cache_database': makes a snapshot of the current database
+- 'uncache_database': forces the next '(time|memory)_start' to rebuild the
+ database.
+
+Scripts are run in the order specified in notmuch-perf-test. In the
+future this order might be chosen automatically so please follow the
+convention of starting the name with 'T' or 'M' followed by two digits
+to specify the order.
projects / notmuch / blobdiff