X-Git-Url: https://git.notmuchmail.org/git?p=notmuch;a=blobdiff_plain;f=test%2FREADME;h=43656a35baf9f9bc5e5d41aca2b65c8e4e763450;hp=f9ac6073d97b4739c80b8ba5ec6da00a88d0e154;hb=260975e8aff635b6b99db835ee8d40ec33ee916c;hpb=a31ad0bd01570307d18b7fbd56b7bcdfb22e5b63

diff --git a/test/README b/test/README
index f9ac6073..43656a35 100644
--- a/test/README
+++ b/test/README
@@ -6,6 +6,19 @@ When fixing bugs or enhancing notmuch, you are strongly encouraged to
 add tests in this directory to cover what you are trying to fix or
 enhance.
 
+Prerequisites
+-------------
+Some tests require external dependencies to run. Without them, they
+will be skipped, or (rarely) marked failed. Please install these, so
+that you know if you break anything.
+
+  - dtach(1)
+  - emacs(1)
+  - emacsclient(1)
+  - gdb(1)
+  - gpg(1)
+  - python(1)
+
 Running Tests
 -------------
 The easiest way to run tests is to say "make test", (or simply run the
@@ -13,7 +26,8 @@ notmuch-test script). Either command will run all available tests.
 
 Alternately, you can run a specific subset of tests by simply invoking
 one of the executable scripts in this directory, (such as ./search,
-./reply, etc.)
+./reply, etc). Note that you will probably want "make test-binaries"
+before running individual tests.
 
 The following command-line options are available when running tests:
 
@@ -55,6 +69,13 @@ can be specified as follows:
 
 	make test OPTIONS="--verbose"
 
+You can choose an emacs binary to run the tests in one of the
+following ways.
+
+	TEST_EMACS=my-special-emacs make test
+	TEST_EMACS=my-special-emacs ./emacs
+	make test TEST_EMACS=my-special-emacs
+
 Skipping Tests
 --------------
 If, for any reason, you need to skip one or more tests, you can do so
@@ -132,20 +153,19 @@ library for your script to use.
    <script>.  If it yields success, test is considered
    successful.  <message> should state what it is testing.
 
- test_expect_failure <message> <script>
-
-   This is NOT the opposite of test_expect_success, but is used
-   to mark a test that demonstrates a known breakage.  Unlike
-   the usual test_expect_success tests, which say "ok" on
-   success and "FAIL" on failure, this will say "FIXED" on
-   success and "still broken" on failure.  Failures from these
-   tests won't cause -i (immediate) to stop.
-
  test_begin_subtest <message>
 
    Set the test description message for a subsequent test_expect_equal
    invocation (see below).
 
+ test_subtest_known_broken
+
+   Mark the current test as broken.  Such tests are expected to fail.
+   Unlike the normal tests, which say "PASS" on success and "FAIL" on
+   failure, these will say "FIXED" on success and "BROKEN" on failure.
+   Failures from these tests won't cause -i (immediate) to stop.  A
+   test must call this before any test_expect_* function.
+
  test_expect_equal <output> <expected>
 
    This is an often-used convenience function built on top of
@@ -163,13 +183,6 @@ library for your script to use.
    robust method to compare formatted textual information, since it
    also notices whitespace and closing newline differences.
 
- test_expect_equal_failure <output> <expected>
-
-   This works similar to test_expect_equal (see above) but is used to
-   mark a test that demonstrates a known breakage, (that is, the
-   author of the test expects "output" and "expected" to differ until
-   the breakage is fixed). See test_expect_failure for details.
-
  test_debug <script>
 
    This takes a single argument, <script>, and evaluates it only
@@ -181,9 +194,21 @@ library for your script to use.
 
    This function executes the provided emacs lisp script within
    emacs. The script can be a sequence of emacs lisp expressions,
-   (that is, they will be evaluated within a progn form). The lisp
-   expressions can call `message' to generate output on stdout to be
-   examined by the calling test script.
+   (that is, they will be evaluated within a progn form). Emacs
+   stdout and stderr is not available, the common way to get output
+   is to save it to a file. There are some auxiliary functions
+   useful in emacs tests provided in test-lib.el. Do not use `setq'
+   for setting variables in Emacs tests because it affects other
+   tests that may run in the same Emacs instance.  Use `let' instead
+   so the scope of the changed variables is limited to a single test.
+
+ test_emacs_expect_t <emacs-lisp-expressions>
+
+  This function executes the provided emacs lisp script within
+  emacs in a manner similar to 'test_emacs'. The expressions should
+  return the value `t' to indicate that the test has passed. If the
+  test does not return `t' then it is considered failed and all data
+  returned by the test is reported to the tester.
 
  test_done
 
@@ -191,8 +216,8 @@ library for your script to use.
    is to summarize successes and failures in the test script and
    exit with an appropriate error code.
 
-There are also a number of mail-specific functions which are useful in
-writing tests:
+There are also a number of notmuch-specific auxiliary functions and
+variables which are useful in writing tests:
 
   generate_message
 
@@ -216,3 +241,15 @@ writing tests:
     will initialize the mail database to a known state of 50 sample
     messages, (culled from the early history of the notmuch mailing
     list).
+
+  notmuch_counter_reset
+  $notmuch_counter_command
+  notmuch_counter_value
+
+    These allow to count how many times notmuch binary is called.
+    notmuch_counter_reset() function generates a script that counts
+    how many times it is called and resets the counter to zero.  The
+    function sets $notmuch_counter_command variable to the path to the
+    generated script that should be called instead of notmuch to do
+    the counting.  The notmuch_counter_value() function prints the
+    current counter value.