X-Git-Url: https://git.notmuchmail.org/git?p=notmuch;a=blobdiff_plain;f=test%2FREADME;h=104a120ea28bfb7250243f1d331ed8efd1e99fa0;hp=79a9b1b2f9a15aba7466e4e5b1b308663dd1296c;hb=b183f2635eaf323066f26c7ca22d931d4f4ccd3a;hpb=47b9314eeecc7ae6c97a5933c8fe028fb6d2b410

diff --git a/test/README b/test/README
index 79a9b1b2..104a120e 100644
--- a/test/README
+++ b/test/README
@@ -8,10 +8,17 @@ enhance.
 
 Prerequisites
 -------------
+The test system itself requires:
+
+  - bash(1) version 4.0 or newer
+
+Without bash 4.0+ the tests just refuse to run.
+
 Some tests require external dependencies to run. Without them, they
 will be skipped, or (rarely) marked failed. Please install these, so
 that you know if you break anything.
 
+  - GNU tar(1)
   - dtach(1)
   - emacs(1)
   - emacsclient(1)
@@ -19,14 +26,21 @@ that you know if you break anything.
   - gpg(1)
   - python(1)
 
+If your system lacks these tools or have older, non-upgreable versions
+of these, please (possibly compile and) install these to some other
+path, for example /usr/local/bin or /opt/gnu/bin. Then prepend the
+chosen directory to your PATH before running the tests.
+
+e.g. env PATH=/opt/gnu/bin:$PATH make test
+
 Running Tests
 -------------
 The easiest way to run tests is to say "make test", (or simply run the
 notmuch-test script). Either command will run all available tests.
 
 Alternately, you can run a specific subset of tests by simply invoking
-one of the executable scripts in this directory, (such as ./search,
-./reply, etc). Note that you will probably want "make test-binaries"
+one of the executable scripts in this directory, (such as ./T*-search.sh,
+./T*-reply.sh, etc). Note that you will probably want "make test-binaries"
 before running individual tests.
 
 The following command-line options are available when running tests:
@@ -64,6 +78,14 @@ The following command-line options are available when running tests:
 	Pointing this argument at a tmpfs filesystem can improve the
 	speed of the test suite for some users.
 
+Certain tests require precomputed databases to complete. You can fetch these
+databases with
+
+	make download-test-databases
+
+If you do not download the test databases, the relevant tests will be
+skipped.
+
 When invoking the test suite via "make test" any of the above options
 can be specified as follows:
 
@@ -72,9 +94,14 @@ can be specified as follows:
 You can choose an emacs binary (and corresponding emacsclient) to run
 the tests in one of the following ways.
 
-	TEST_EMACS=my-special-emacs TEST_EMACSCLIENT=my-emacsclient make test
-	TEST_EMACS=my-special-emacs TEST_EMACSCLIENT=my-emacsclient ./emacs
-	make test TEST_EMACS=my-special-emacs TEST_EMACSCLIENT=my-emacsclient
+	TEST_EMACS=my-emacs TEST_EMACSCLIENT=my-emacsclient make test
+	TEST_EMACS=my-emacs TEST_EMACSCLIENT=my-emacsclient ./T*-emacs.sh
+	make test TEST_EMACS=my-emacs TEST_EMACSCLIENT=my-emacsclient
+
+Some tests may require a c compiler. You can choose the name and flags similarly
+to with emacs, e.g.
+
+     make test TEST_CC=gcc TEST_CFLAGS="-g -O2"
 
 Quiet Execution
 ---------------
@@ -104,16 +131,23 @@ Note that some tests in the existing test suite rely on previous test
 items, so you cannot arbitrarily skip any test and expect the
 remaining tests to be unaffected.
 
+Currently we do not consider skipped tests as build failures. For
+maximum robustness, when setting up automated build processes, you
+should explicitely skip tests, rather than relying on notmuch's
+detection of missing prerequisites. In the future we may treat tests
+unable to run because of missing prerequisites, but not explicitely
+skipped by the user, as failures.
+
 Writing Tests
 -------------
-The test script is written as a shell script.  It should start with
-the standard "#!/usr/bin/env bash" with copyright notices, and an
-assignment to variable 'test_description', like this:
+The test script is written as a shell script. It is to be named as
+Tddd-testname.sh where 'ddd' is three digits and 'testname' the "bare"
+name of your test. Tests will be run in order the 'ddd' part determines.
+
+The test script should start with the standard "#!/usr/bin/env bash"
+and an assignment to variable 'test_description', like this:
 
 	#!/usr/bin/env bash
-	#
-	# Copyright (c) 2005 Junio C Hamano
-	#
 
 	test_description='xxx test (option --frotz)
 
@@ -125,7 +159,7 @@ Source 'test-lib.sh'
 After assigning test_description, the test script should source
 test-lib.sh like this:
 
-	. ./test-lib.sh
+	. ./test-lib.sh || exit 1
 
 This test harness library does the following things: