X-Git-Url: https://git.notmuchmail.org/git?p=notmuch;a=blobdiff_plain;f=performance-test%2FREADME;h=fbc61028d2db4dfdc204cd4a78e2613cadd78c3c;hp=36f1dacd3cbff9e35c78801fe8712e37fe3e72a2;hb=HEAD;hpb=80639761b5242781d06073eec5721679051a4a3a

diff --git a/performance-test/README b/performance-test/README
index 36f1dacd..1ca0df2f 100644
--- a/performance-test/README
+++ b/performance-test/README
@@ -1,3 +1,10 @@
+Performance Tests
+-----------------
+
+This directory contains two kinds of performance tests: time tests,
+and memory tests. The former use gnu time, and the latter use
+valgrind.
+
 Pre-requisites
 --------------
 
@@ -5,9 +12,11 @@ In addition to having notmuch, you need:
 
 - gpg
 - gnu tar
-- gnu time
+- gnu time (for the time tests)
 - xz. Some speedup can be gotten by installing "pixz", but this is
   probably only worthwhile if you are debugging the tests.
+- valgrind (for the memory tests)
+- perf (optional, for more fine-grained timing)
 
 Getting set up to run tests:
 ----------------------------
@@ -15,11 +24,11 @@ Getting set up to run tests:
 First, you need to get the corpus.  If you don't already have the gpg
 key for David Bremner, run
 
-   % gpg --search 'david@tethera.net'
+   % gpg --locate-external-key 'david@tethera.net'
 
 This should get you a key with fingerprint
 
-    815B 6398 2A79 F8E7 C727  86C4 762B 57BB 7842 06AD
+    7A18 807F 100A 4570 C596  8420 7E4E 65C8 720B 706B
 
 (the last 8 digits are printed as the "key id").
 
@@ -29,30 +38,67 @@ To fetch the actual corpus it should work to run
 
 In case that fails or is too slow, check
 
-   http://notmuchmail.org/corpus
+   https://notmuchmail.org/corpus
 
 for a list of mirrors.
 
 Running tests
 -------------
 
-The easiest way to run performance tests is to say "make perf-test", (or
-simply run the notmuch-perf-test script). Either command will run all
-available performance tests.
+The easiest way to run performance tests is to say "make perf-test".
+This will run all time and memory tests.  Be aware that the memory
+tests are quite time consuming when run on the full corpus, and that
+depending on your interests it may be more sensible to run "make
+time-test" or "make memory-test".  You can also invoke one of the
+scripts notmuch-time-test or notmuch-memory-test or run a more
+specific subset of tests by simply invoking one of the executable
+scripts in this directory, (such as ./T00-new).  Each test script
+supports the following arguments
+
+--small / --medium / --large	Choose corpus size.
+--debug				Enable debugging. In particular don't delete
+                                temporary directories.
+--perf                          Run perf record in place of /usr/bin/time. Perf output can be
+                                found in a log directory.
+--call-graph {fp,lbr,dwarf}     Call graph option for perf record. Default is 'lbr'.
+
+When using the make targets, you can pass arguments to all test
+scripts by defining the make variable OPTIONS.
+
+Log Directory
+-------------
+
+The memory tests, and the time tests when option '--perf' is given
+save their output in a directory named as follows
 
-Alternately, you can run a specific subset of tests by simply invoking
-one of the executable scripts in this directory, (such as ./basic).
+     log.$test_name-$corpus_size-$timestamp
+
+These directories are removed by "make clean".
 
 Writing tests
 -------------
 
-Have a look at "basic" for an example.
+Have a look at "T01-dump-restore" for an example time test and
+"M00-new" for an example memory test. In both cases sourcing
+"perf-test-lib.sh" is mandatory.
+
+Basics:
+
+- '(time|memory)_start' unpacks the mail corpus and calls notmuch new if it
+   cannot find a cache of the appropriate corpus.
+- '(time|memory)_run' runs the command under time or valgrind. Currently
+  "memory_run" does not support i/o redirection in the command.
+- '(time|memory)_done' does the cleanup; comment it out or pass --debug to the
+  script to leave the temporary files around.
 
-add_email_corpus takes arguments "--small" and "--medium" for when you
-want smaller subsets of the corpus to check.
+Utility functions include
 
-time_done does the cleanup; comment it out or define "$debug" to leave
-the temporary files around.
+- 'add_email_corpus' unpacks a set of messages and tags
+- 'cache_database': makes a snapshot of the current database
+- 'uncache_database': forces the next '(time|memory)_start' to rebuild the
+  database.
 
-Currently there is no option processing (e.g. --debug) in the
-performance tests.
+Scripts are run in the order specified in notmuch-perf-test. In the
+future this order might be chosen automatically so please follow the
+convention of starting the name with 'T' or 'M' followed by two digits
+to specify the order.