X-Git-Url: https://git.notmuchmail.org/git?p=notmuch;a=blobdiff_plain;f=test%2FREADME;h=8e06f44241a667609fdd5b47eda05b51e6fd3b47;hp=be75e0e706e7af579c087243dcb01aae3cc6b195;hb=87bdfbc91f65cb1031ef0ac8a804759f2061ac10;hpb=68a2c7a8b0f749cb33a8ce7cfa2aa7781d2529bb

diff --git a/test/README b/test/README
index be75e0e7..8e06f442 100644
--- a/test/README
+++ b/test/README
@@ -6,14 +6,53 @@ When fixing bugs or enhancing notmuch, you are strongly encouraged to
 add tests in this directory to cover what you are trying to fix or
 enhance.
 
+Prerequisites
+-------------
+The test system itself requires:
+
+  - bash(1) version 4.0 or newer
+
+Without bash 4.0+ the tests just refuse to run.
+
+Some tests require external dependencies to run. Without them, they
+will be skipped, or (rarely) marked failed. Please install these, so
+that you know if you break anything.
+
+  - GNU tar(1)
+  - dtach(1)
+  - emacs(1)
+  - emacsclient(1)
+  - gdb(1)
+  - gpg(1)
+  - python(1)
+
+If your system lacks these tools or have older, non-upgradable versions
+of these, please (possibly compile and) install these to some other
+path, for example /usr/local/bin or /opt/gnu/bin. Then prepend the
+chosen directory to your PATH before running the tests.
+
+e.g. env PATH=/opt/gnu/bin:$PATH make test
+
+For FreeBSD you need to install latest gdb from ports or packages and
+provide path to it in TEST_GDB environment variable before executing
+the tests, native FreeBSD gdb does not not work.  If you install
+coreutils, which provides GNU versions of basic utils like 'date' and
+'base64' on FreeBSD, the test suite will use these instead of the
+native ones. This provides robustness against portability issues with
+these system tools. Most often the tests are written, reviewed and
+tested on Linux system so such portability issues arise from time to
+time.
+
+
 Running Tests
 -------------
 The easiest way to run tests is to say "make test", (or simply run the
 notmuch-test script). Either command will run all available tests.
 
 Alternately, you can run a specific subset of tests by simply invoking
-one of the executable scripts in this directory, (such as ./search,
-./reply, etc.)
+one of the executable scripts in this directory, (such as ./T*-search.sh,
+./T*-reply.sh, etc). Note that you will probably want "make test-binaries"
+before running individual tests.
 
 The following command-line options are available when running tests:
 
@@ -41,11 +80,48 @@ The following command-line options are available when running tests:
 	As the names depend on the tests' file names, it is safe to
 	run the tests with this option in parallel.
 
+--root=<dir>::
+	This runs the testsuites specified under a separate directory.
+	However, caution is advised, as not all tests are maintained
+	with this relocation in mind, so some tests may behave
+	differently.
+
+	Pointing this argument at a tmpfs filesystem can improve the
+	speed of the test suite for some users.
+
+Certain tests require precomputed databases to complete. You can fetch these
+databases with
+
+	make download-test-databases
+
+If you do not download the test databases, the relevant tests will be
+skipped.
+
 When invoking the test suite via "make test" any of the above options
 can be specified as follows:
 
 	make test OPTIONS="--verbose"
 
+You can choose an emacs binary (and corresponding emacsclient) to run
+the tests in one of the following ways.
+
+	TEST_EMACS=my-emacs TEST_EMACSCLIENT=my-emacsclient make test
+	TEST_EMACS=my-emacs TEST_EMACSCLIENT=my-emacsclient ./T*-emacs.sh
+	make test TEST_EMACS=my-emacs TEST_EMACSCLIENT=my-emacsclient
+
+Some tests may require a c compiler. You can choose the name and flags similarly
+to with emacs, e.g.
+
+     make test TEST_CC=gcc TEST_CFLAGS="-g -O2"
+
+Quiet Execution
+---------------
+
+Normally, when new script starts and when test PASSes you get a message
+printed on screen. This printing can be disabled by setting the
+NOTMUCH_TEST_QUIET variable to a non-null value. Message on test
+failures and skips are still printed.
+
 Skipping Tests
 --------------
 If, for any reason, you need to skip one or more tests, you can do so
@@ -66,16 +142,23 @@ Note that some tests in the existing test suite rely on previous test
 items, so you cannot arbitrarily skip any test and expect the
 remaining tests to be unaffected.
 
+Currently we do not consider skipped tests as build failures. For
+maximum robustness, when setting up automated build processes, you
+should explicitly skip tests, rather than relying on notmuch's
+detection of missing prerequisites. In the future we may treat tests
+unable to run because of missing prerequisites, but not explicitly
+skipped by the user, as failures.
+
 Writing Tests
 -------------
-The test script is written as a shell script.  It should start with
-the standard "#!/usr/bin/env bash" with copyright notices, and an
-assignment to variable 'test_description', like this:
+The test script is written as a shell script. It is to be named as
+Tddd-testname.sh where 'ddd' is three digits and 'testname' the "bare"
+name of your test. Tests will be run in order the 'ddd' part determines.
+
+The test script should start with the standard "#!/usr/bin/env bash"
+and an assignment to variable 'test_description', like this:
 
 	#!/usr/bin/env bash
-	#
-	# Copyright (c) 2005 Junio C Hamano
-	#
 
 	test_description='xxx test (option --frotz)
 
@@ -87,7 +170,7 @@ Source 'test-lib.sh'
 After assigning test_description, the test script should source
 test-lib.sh like this:
 
-	. ./test-lib.sh
+	. ./test-lib.sh || exit 1
 
 This test harness library does the following things:
 
@@ -117,25 +200,29 @@ Test harness library
 There are a handful helper functions defined in the test harness
 library for your script to use.
 
- test_expect_success <message> <script>
+ test_begin_subtest <message>
+
+   Set the test description message for a subsequent test_expect_*
+   invocation (see below).
+
+ test_expect_success <script>
 
-   This takes two strings as parameter, and evaluates the
+   This takes a string as parameter, and evaluates the
    <script>.  If it yields success, test is considered
-   successful.  <message> should state what it is testing.
+   successful.
 
- test_expect_failure <message> <script>
+ test_expect_code <code> <script>
 
-   This is NOT the opposite of test_expect_success, but is used
-   to mark a test that demonstrates a known breakage.  Unlike
-   the usual test_expect_success tests, which say "ok" on
-   success and "FAIL" on failure, this will say "FIXED" on
-   success and "still broken" on failure.  Failures from these
-   tests won't cause -i (immediate) to stop.
+   This takes two strings as parameter, and evaluates the <script>.
+   If it yields <code> exit status, test is considered successful.
 
- test_begin_subtest <message>
+ test_subtest_known_broken
 
-   Set the test description message for a subsequent test_expect_equal
-   invocation (see below).
+   Mark the current test as broken.  Such tests are expected to fail.
+   Unlike the normal tests, which say "PASS" on success and "FAIL" on
+   failure, these will say "FIXED" on success and "BROKEN" on failure.
+   Failures from these tests won't cause -i (immediate) to stop.  A
+   test must call this before any test_expect_* function.
 
  test_expect_equal <output> <expected>
 
@@ -147,12 +234,19 @@ library for your script to use.
    will generate a failure and print the difference of the two
    strings.
 
- test_expect_equal_failure <output> <expected>
+ test_expect_equal_file <file1> <file2>
+
+   Identical to test_expect_equal, except that <file1> and <file2>
+   are files instead of strings.  This is a much more robust method to
+   compare formatted textual information, since it also notices
+   whitespace and closing newline differences.
 
-   This works similar to test_expect_equal (see above) but is used to
-   mark a test that demonstrates a known breakage, (that is, the
-   author of the test expects "output" and "expected" to differ until
-   the breakage is fixed). See test_expect_failure for details.
+ test_expect_equal_json <output> <expected>
+
+   Identical to test_expect_equal, except that the two strings are
+   treated as JSON and canonicalized before equality testing.  This is
+   useful to abstract away from whitespace differences in the expected
+   output and that generated by running a notmuch command.
 
  test_debug <script>
 
@@ -165,9 +259,21 @@ library for your script to use.
 
    This function executes the provided emacs lisp script within
    emacs. The script can be a sequence of emacs lisp expressions,
-   (that is, they will be evaluated within a progn form). The lisp
-   expressions can call `message' to generate output on stdout to be
-   examined by the calling test script.
+   (that is, they will be evaluated within a progn form). Emacs
+   stdout and stderr is not available, the common way to get output
+   is to save it to a file. There are some auxiliary functions
+   useful in emacs tests provided in test-lib.el. Do not use `setq'
+   for setting variables in Emacs tests because it affects other
+   tests that may run in the same Emacs instance.  Use `let' instead
+   so the scope of the changed variables is limited to a single test.
+
+ test_emacs_expect_t <emacs-lisp-expressions>
+
+  This function executes the provided emacs lisp script within
+  emacs in a manner similar to 'test_emacs'. The expressions should
+  return the value `t' to indicate that the test has passed. If the
+  test does not return `t' then it is considered failed and all data
+  returned by the test is reported to the tester.
 
  test_done
 
@@ -175,8 +281,8 @@ library for your script to use.
    is to summarize successes and failures in the test script and
    exit with an appropriate error code.
 
-There are also a number of mail-specific functions which are useful in
-writing tests:
+There are also a number of notmuch-specific auxiliary functions and
+variables which are useful in writing tests:
 
   generate_message
 
@@ -200,3 +306,28 @@ writing tests:
     will initialize the mail database to a known state of 50 sample
     messages, (culled from the early history of the notmuch mailing
     list).
+
+  notmuch_counter_reset
+  $notmuch_counter_command
+  notmuch_counter_value
+
+    These allow to count how many times notmuch binary is called.
+    notmuch_counter_reset() function generates a script that counts
+    how many times it is called and resets the counter to zero.  The
+    function sets $notmuch_counter_command variable to the path to the
+    generated script that should be called instead of notmuch to do
+    the counting.  The notmuch_counter_value() function prints the
+    current counter value.
+
+There are also functions which remove various environment-dependent
+values from notmuch output; these are useful to ensure that test
+results remain consistent across different machines.
+
+ notmuch_search_sanitize
+ notmuch_show_sanitize
+ notmuch_show_sanitize_all
+ notmuch_json_show_sanitize
+
+   All these functions should receive the text to be sanitized as the
+   input of a pipe, e.g.
+   output=`notmuch search "..." | notmuch_search_sanitize`