perf-test: initial version of memory test infrastructure.

[notmuch] / performance-test / README

diff --git a/performance-test/README b/performance-test/README
index d1fb6de4cfa742942998045b5b0da7a8c5a70b20..996724cdf9472b0cb110446402d97e199668ca0e 100644 (file)
--- a/performance-test/README
+++ b/performance-test/README
@@ -1,3 +1,10 @@
+Performance Tests
+-----------------
+
+This directory contains two kinds of performance tests: time tests,
+and memory tests. The former use gnu time, and the latter use
+valgrind.
+

 Pre-requisites
 --------------
 
 Pre-requisites
 --------------
 


@@ -5,9 +12,10 @@ In addition to having notmuch, you need:
 
 - gpg
 - gnu tar
 
 - gpg
 - gnu tar

-- gnu time
+- gnu time (for the time tests)

 - xz. Some speedup can be gotten by installing "pixz", but this is
   probably only worthwhile if you are debugging the tests.
 - xz. Some speedup can be gotten by installing "pixz", but this is
   probably only worthwhile if you are debugging the tests.

+- valgrind (for the memory tests)

 
 Getting set up to run tests:
 ----------------------------
 
 Getting set up to run tests:
 ----------------------------


@@ -36,34 +44,47 @@ for a list of mirrors.
 Running tests
 -------------
 
 Running tests
 -------------
 

-The easiest way to run performance tests is to say "make time-test", (or
-simply run the notmuch-time-test script). Either command will run all
-available performance tests.
-
-Alternately, you can run a specific subset of tests by simply invoking
-one of the executable scripts in this directory, (such as ./basic).
-Each test script supports the following arguments
+The easiest way to run performance tests is to say "make perf-test".
+This will run all time and memory tests.  Be aware that the memory
+tests are quite time consuming when run on the full corpus, and that
+depending on your interests it may be more sensible to run "make
+time-test" or "make memory-test".  You can also invoke one of the
+scripts notmuch-time-test or notmuch-memory-test or run a more
+specific subset of tests by simply invoking one of the executable
+scripts in this directory, (such as ./T00-new).  Each test script
+supports the following arguments

 
 --small / --medium / --large   Choose corpus size.
 --debug                                Enable debugging. In particular don't delete
                                temporary directories.
 
 
 --small / --medium / --large   Choose corpus size.
 --debug                                Enable debugging. In particular don't delete
                                temporary directories.
 

+When using the make targets, you can pass arguments to all test
+scripts by defining the make variable OPTIONS.
+

 Writing tests
 -------------
 
 Writing tests
 -------------
 

-Have a look at "T01-dump-restore" for an example. Sourcing
-"perf-test-lib.sh" is mandatory.  Utility functions include
+Have a look at "T01-dump-restore" for an example time test and
+"M00-new" for an example memory test. In both cases sourcing
+"perf-test-lib.sh" is mandatory.

 
 

-- 'add_email_corpus' unpacks a set of messages and adds them to the database.
-- 'cache_database': makes a snapshot of the current database
-- 'uncache_database': forces the next 'add_email_corpus' to rebuild the
-  database.
-- 'time_start' unpacks the mail corpus and calls notmuch new if it
+Basics:
+
+- '(time|memory)_start' unpacks the mail corpus and calls notmuch new if it

    cannot find a cache of the appropriate corpus.
    cannot find a cache of the appropriate corpus.

-- 'time_done' does the cleanup; comment it out or pass --debug to the
+- '(time|memory)_run' runs the command under time or valgrind. Currently
+  "memory_run" does not support i/o redirection in the command.
+- '(time|memory)_done' does the cleanup; comment it out or pass --debug to the

   script to leave the temporary files around.
 
   script to leave the temporary files around.
 

+Utility functions include
+
+- 'add_email_corpus' unpacks a set of messages and tags
+- 'cache_database': makes a snapshot of the current database
+- 'uncache_database': forces the next '(time|memory)_start' to rebuild the
+  database.
+

 Scripts are run in the order specified in notmuch-perf-test. In the
 future this order might be chosen automatically so please follow the
 Scripts are run in the order specified in notmuch-perf-test. In the
 future this order might be chosen automatically so please follow the

-convention of starting the name with 'T' followed by two digits to
-specify the order.
+convention of starting the name with 'T' or 'M' followed by two digits
+to specify the order.