X-Git-Url: https://git.notmuchmail.org/git?p=notmuch;a=blobdiff_plain;f=test%2FREADME;h=79a9b1b2f9a15aba7466e4e5b1b308663dd1296c;hp=2481f16d1ebefac322f855ef2c8b043a17988185;hb=84719b08f757a6079f4c3331d0c476d19b265948;hpb=3ee541ab3184a56a5fbe194caf0ff1f15e124683

diff --git a/test/README b/test/README
index 2481f16d..79a9b1b2 100644
--- a/test/README
+++ b/test/README
@@ -6,6 +6,19 @@ When fixing bugs or enhancing notmuch, you are strongly encouraged to
 add tests in this directory to cover what you are trying to fix or
 enhance.
 
+Prerequisites
+-------------
+Some tests require external dependencies to run. Without them, they
+will be skipped, or (rarely) marked failed. Please install these, so
+that you know if you break anything.
+
+  - dtach(1)
+  - emacs(1)
+  - emacsclient(1)
+  - gdb(1)
+  - gpg(1)
+  - python(1)
+
 Running Tests
 -------------
 The easiest way to run tests is to say "make test", (or simply run the
@@ -13,7 +26,8 @@ notmuch-test script). Either command will run all available tests.
 
 Alternately, you can run a specific subset of tests by simply invoking
 one of the executable scripts in this directory, (such as ./search,
-./reply, etc.)
+./reply, etc). Note that you will probably want "make test-binaries"
+before running individual tests.
 
 The following command-line options are available when running tests:
 
@@ -42,7 +56,7 @@ The following command-line options are available when running tests:
 	run the tests with this option in parallel.
 
 --root=<dir>::
-	This runs the testsuites specified under a seperate directory.
+	This runs the testsuites specified under a separate directory.
 	However, caution is advised, as not all tests are maintained
 	with this relocation in mind, so some tests may behave
 	differently.
@@ -55,6 +69,21 @@ can be specified as follows:
 
 	make test OPTIONS="--verbose"
 
+You can choose an emacs binary (and corresponding emacsclient) to run
+the tests in one of the following ways.
+
+	TEST_EMACS=my-special-emacs TEST_EMACSCLIENT=my-emacsclient make test
+	TEST_EMACS=my-special-emacs TEST_EMACSCLIENT=my-emacsclient ./emacs
+	make test TEST_EMACS=my-special-emacs TEST_EMACSCLIENT=my-emacsclient
+
+Quiet Execution
+---------------
+
+Normally, when new script starts and when test PASSes you get a message
+printed on screen. This printing can be disabled by setting the
+NOTMUCH_TEST_QUIET variable to a non-null value. Message on test
+failures and skips are still printed.
+
 Skipping Tests
 --------------
 If, for any reason, you need to skip one or more tests, you can do so
@@ -155,12 +184,19 @@ library for your script to use.
    will generate a failure and print the difference of the two
    strings.
 
- test_expect_equal_file <output> <expected>
+ test_expect_equal_file <file1> <file2>
+
+   Identical to test_expect_equal, except that <file1> and <file2>
+   are files instead of strings.  This is a much more robust method to
+   compare formatted textual information, since it also notices
+   whitespace and closing newline differences.
+
+ test_expect_equal_json <output> <expected>
 
-   Identical to test_exepect_equal, except that <output> and
-   <expected> are files instead of strings.  This is a much more
-   robust method to compare formatted textual information, since it
-   also notices whitespace and closing newline differences.
+   Identical to test_expect_equal, except that the two strings are
+   treated as JSON and canonicalized before equality testing.  This is
+   useful to abstract away from whitespace differences in the expected
+   output and that generated by running a notmuch command.
 
  test_debug <script>
 
@@ -181,14 +217,22 @@ library for your script to use.
    tests that may run in the same Emacs instance.  Use `let' instead
    so the scope of the changed variables is limited to a single test.
 
+ test_emacs_expect_t <emacs-lisp-expressions>
+
+  This function executes the provided emacs lisp script within
+  emacs in a manner similar to 'test_emacs'. The expressions should
+  return the value `t' to indicate that the test has passed. If the
+  test does not return `t' then it is considered failed and all data
+  returned by the test is reported to the tester.
+
  test_done
 
    Your test script must have test_done at the end.  Its purpose
    is to summarize successes and failures in the test script and
    exit with an appropriate error code.
 
-There are also a number of mail-specific functions which are useful in
-writing tests:
+There are also a number of notmuch-specific auxiliary functions and
+variables which are useful in writing tests:
 
   generate_message
 
@@ -212,3 +256,28 @@ writing tests:
     will initialize the mail database to a known state of 50 sample
     messages, (culled from the early history of the notmuch mailing
     list).
+
+  notmuch_counter_reset
+  $notmuch_counter_command
+  notmuch_counter_value
+
+    These allow to count how many times notmuch binary is called.
+    notmuch_counter_reset() function generates a script that counts
+    how many times it is called and resets the counter to zero.  The
+    function sets $notmuch_counter_command variable to the path to the
+    generated script that should be called instead of notmuch to do
+    the counting.  The notmuch_counter_value() function prints the
+    current counter value.
+
+There are also functions which remove various environment-dependent
+values from notmuch output; these are useful to ensure that test
+results remain consistent across different machines.
+
+ notmuch_search_sanitize
+ notmuch_show_sanitize
+ notmuch_show_sanitize_all
+ notmuch_json_show_sanitize
+
+   All these functions should receive the text to be sanitized as the
+   input of a pipe, e.g.
+   output=`notmuch search "..." | notmuch_search_sanitize`