test suite: don't consider skipped individual tests as failing

[notmuch] / test / README

diff --git a/test/README b/test/README
index 79a9b1b2f9a15aba7466e4e5b1b308663dd1296c..e54e36b795e852e536dab96256420edbfee84368 100644 (file)
--- a/test/README
+++ b/test/README
@@ -64,6 +64,14 @@ The following command-line options are available when running tests:
        Pointing this argument at a tmpfs filesystem can improve the
        speed of the test suite for some users.
 
+Certain tests require precomputed databases to complete. You can fetch these
+databases with
+
+       make download-test-databases
+
+If you do not download the test databases, the relevant tests will be
+skipped.
+
 When invoking the test suite via "make test" any of the above options
 can be specified as follows:
 
@@ -76,6 +84,11 @@ the tests in one of the following ways.
        TEST_EMACS=my-special-emacs TEST_EMACSCLIENT=my-emacsclient ./emacs
        make test TEST_EMACS=my-special-emacs TEST_EMACSCLIENT=my-emacsclient
 
+Some tests may require a c compiler. You can choose the name and flags similarly
+to with emacs, e.g.
+
+     make test TEST_CC=gcc TEST_CFLAGS="-g -O2"
+
 Quiet Execution
 ---------------
 
@@ -104,6 +117,13 @@ Note that some tests in the existing test suite rely on previous test
 items, so you cannot arbitrarily skip any test and expect the
 remaining tests to be unaffected.
 
+Currently we do not consider skipped tests as build failures. For
+maximum robustness, when setting up automated build processes, you
+should explicitely skip tests, rather than relying on notmuch's
+detection of missing prerequisites. In the future we may treat tests
+unable to run because of missing prerequisites, but not explicitely
+skipped by the user, as failures.
+
 Writing Tests
 -------------
 The test script is written as a shell script.  It should start with
@@ -125,7 +145,7 @@ Source 'test-lib.sh'
 After assigning test_description, the test script should source
 test-lib.sh like this:
 
-       . ./test-lib.sh
+       . ./test-lib.sh || exit 1
 
 This test harness library does the following things: