emacs: Add new option notmuch-search-hide-excluded

[notmuch] / performance-test / README

diff --git a/performance-test/README b/performance-test/README
index d1fb6de4cfa742942998045b5b0da7a8c5a70b20..1ca0df2f664c9e711a6ec0977ff76b8fce60036e 100644 (file)
--- a/performance-test/README
+++ b/performance-test/README
@@ -1,3 +1,10 @@
+Performance Tests
+-----------------
+
+This directory contains two kinds of performance tests: time tests,
+and memory tests. The former use gnu time, and the latter use
+valgrind.
+

 Pre-requisites
 --------------
 
 Pre-requisites
 --------------
 


@@ -5,9 +12,11 @@ In addition to having notmuch, you need:
 
 - gpg
 - gnu tar
 
 - gpg
 - gnu tar

-- gnu time
+- gnu time (for the time tests)

 - xz. Some speedup can be gotten by installing "pixz", but this is
   probably only worthwhile if you are debugging the tests.
 - xz. Some speedup can be gotten by installing "pixz", but this is
   probably only worthwhile if you are debugging the tests.

+- valgrind (for the memory tests)
+- perf (optional, for more fine-grained timing)

 
 Getting set up to run tests:
 ----------------------------
 
 Getting set up to run tests:
 ----------------------------


@@ -15,11 +24,11 @@ Getting set up to run tests:
 First, you need to get the corpus.  If you don't already have the gpg
 key for David Bremner, run
 
 First, you need to get the corpus.  If you don't already have the gpg
 key for David Bremner, run
 

-   % gpg --search 'david@tethera.net'
+   % gpg --locate-external-key 'david@tethera.net'

 
 This should get you a key with fingerprint
 
 
 This should get you a key with fingerprint
 

-    815B 6398 2A79 F8E7 C727  86C4 762B 57BB 7842 06AD
+    7A18 807F 100A 4570 C596  8420 7E4E 65C8 720B 706B

 
 (the last 8 digits are printed as the "key id").
 
 
 (the last 8 digits are printed as the "key id").
 


@@ -29,41 +38,67 @@ To fetch the actual corpus it should work to run
 
 In case that fails or is too slow, check
 
 
 In case that fails or is too slow, check
 

-   http://notmuchmail.org/corpus
+   https://notmuchmail.org/corpus

 
 for a list of mirrors.
 
 Running tests
 -------------
 
 
 for a list of mirrors.
 
 Running tests
 -------------
 

-The easiest way to run performance tests is to say "make time-test", (or
-simply run the notmuch-time-test script). Either command will run all
-available performance tests.
-
-Alternately, you can run a specific subset of tests by simply invoking
-one of the executable scripts in this directory, (such as ./basic).
-Each test script supports the following arguments
+The easiest way to run performance tests is to say "make perf-test".
+This will run all time and memory tests.  Be aware that the memory
+tests are quite time consuming when run on the full corpus, and that
+depending on your interests it may be more sensible to run "make
+time-test" or "make memory-test".  You can also invoke one of the
+scripts notmuch-time-test or notmuch-memory-test or run a more
+specific subset of tests by simply invoking one of the executable
+scripts in this directory, (such as ./T00-new).  Each test script
+supports the following arguments

 
 --small / --medium / --large   Choose corpus size.
 --debug                                Enable debugging. In particular don't delete
 
 --small / --medium / --large   Choose corpus size.
 --debug                                Enable debugging. In particular don't delete

-                               temporary directories.
+                                temporary directories.
+--perf                          Run perf record in place of /usr/bin/time. Perf output can be
+                                found in a log directory.
+--call-graph {fp,lbr,dwarf}     Call graph option for perf record. Default is 'lbr'.
+
+When using the make targets, you can pass arguments to all test
+scripts by defining the make variable OPTIONS.
+
+Log Directory
+-------------
+
+The memory tests, and the time tests when option '--perf' is given
+save their output in a directory named as follows
+
+     log.$test_name-$corpus_size-$timestamp
+
+These directories are removed by "make clean".

 
 Writing tests
 -------------
 
 
 Writing tests
 -------------
 

-Have a look at "T01-dump-restore" for an example. Sourcing
-"perf-test-lib.sh" is mandatory.  Utility functions include
+Have a look at "T01-dump-restore" for an example time test and
+"M00-new" for an example memory test. In both cases sourcing
+"perf-test-lib.sh" is mandatory.

 
 

-- 'add_email_corpus' unpacks a set of messages and adds them to the database.
-- 'cache_database': makes a snapshot of the current database
-- 'uncache_database': forces the next 'add_email_corpus' to rebuild the
-  database.
-- 'time_start' unpacks the mail corpus and calls notmuch new if it
+Basics:
+
+- '(time|memory)_start' unpacks the mail corpus and calls notmuch new if it

    cannot find a cache of the appropriate corpus.
    cannot find a cache of the appropriate corpus.

-- 'time_done' does the cleanup; comment it out or pass --debug to the
+- '(time|memory)_run' runs the command under time or valgrind. Currently
+  "memory_run" does not support i/o redirection in the command.
+- '(time|memory)_done' does the cleanup; comment it out or pass --debug to the

   script to leave the temporary files around.
 
   script to leave the temporary files around.
 

+Utility functions include
+
+- 'add_email_corpus' unpacks a set of messages and tags
+- 'cache_database': makes a snapshot of the current database
+- 'uncache_database': forces the next '(time|memory)_start' to rebuild the
+  database.
+

 Scripts are run in the order specified in notmuch-perf-test. In the
 future this order might be chosen automatically so please follow the
 Scripts are run in the order specified in notmuch-perf-test. In the
 future this order might be chosen automatically so please follow the

-convention of starting the name with 'T' followed by two digits to
-specify the order.
+convention of starting the name with 'T' or 'M' followed by two digits
+to specify the order.