emacs: Add new option notmuch-search-hide-excluded

[notmuch] / test / README

diff --git a/test/README b/test/README
index f9ac6073d97b4739c80b8ba5ec6da00a88d0e154..a81808b164f0cec0ec6c0010cb023fe77d7e098c 100644 (file)
--- a/test/README
+++ b/test/README
@@ -6,14 +6,53 @@ When fixing bugs or enhancing notmuch, you are strongly encouraged to
 add tests in this directory to cover what you are trying to fix or
 enhance.
 
 add tests in this directory to cover what you are trying to fix or
 enhance.
 

+Prerequisites
+-------------
+The test system itself requires:
+
+  - bash(1) version 4.0 or newer
+
+Without bash 4.0+ the tests just refuse to run.
+
+Some tests require external dependencies to run. Without them, they
+will be skipped, or (rarely) marked failed. Please install these, so
+that you know if you break anything.
+
+  - GNU tar(1)
+  - dtach(1)
+  - emacs(1)
+  - emacsclient(1)
+  - gdb(1)
+  - gpg(1)
+  - python(1)
+  - xapian-metadata(1)
+
+If your system lacks these tools or have older, non-upgradable versions
+of these, please (possibly compile and) install these to some other
+path, for example /usr/local/bin or /opt/gnu/bin. Then prepend the
+chosen directory to your PATH before running the tests.
+
+e.g. env PATH=/opt/gnu/bin:$PATH make test
+
+For FreeBSD you need to install latest gdb from ports or packages and
+provide path to it in TEST_GDB environment variable before executing
+the tests, native FreeBSD gdb does not not work.  If you install
+coreutils, which provides GNU versions of basic utils like 'date' and
+'base64' on FreeBSD, the test suite will use these instead of the
+native ones. This provides robustness against portability issues with
+these system tools. Most often the tests are written, reviewed and
+tested on Linux system so such portability issues arise from time to
+time.
+

 Running Tests
 -------------
 The easiest way to run tests is to say "make test", (or simply run the
 notmuch-test script). Either command will run all available tests.
 
 Alternately, you can run a specific subset of tests by simply invoking
 Running Tests
 -------------
 The easiest way to run tests is to say "make test", (or simply run the
 notmuch-test script). Either command will run all available tests.
 
 Alternately, you can run a specific subset of tests by simply invoking

-one of the executable scripts in this directory, (such as ./search,
-./reply, etc.)
+one of the executable scripts in this directory, (such as ./T*-search.sh,
+./T*-reply.sh, etc). Note that you will probably want "make test-binaries"
+before running individual tests.

 
 The following command-line options are available when running tests:
 
 
 The following command-line options are available when running tests:
 


@@ -41,20 +80,36 @@ The following command-line options are available when running tests:
        As the names depend on the tests' file names, it is safe to
        run the tests with this option in parallel.
 
        As the names depend on the tests' file names, it is safe to
        run the tests with this option in parallel.
 

---root=<dir>::
-       This runs the testsuites specified under a seperate directory.
-       However, caution is advised, as not all tests are maintained
-       with this relocation in mind, so some tests may behave
-       differently.
-
-       Pointing this argument at a tmpfs filesystem can improve the
-       speed of the test suite for some users.
-

 When invoking the test suite via "make test" any of the above options
 can be specified as follows:
 
        make test OPTIONS="--verbose"
 
 When invoking the test suite via "make test" any of the above options
 can be specified as follows:
 
        make test OPTIONS="--verbose"
 

+You can choose an emacs binary (and corresponding emacsclient) to run
+the tests in one of the following ways.
+
+       TEST_EMACS=my-emacs TEST_EMACSCLIENT=my-emacsclient make test
+       TEST_EMACS=my-emacs TEST_EMACSCLIENT=my-emacsclient ./T*-emacs.sh
+       make test TEST_EMACS=my-emacs TEST_EMACSCLIENT=my-emacsclient
+
+Some tests may require a c compiler. You can choose the name and flags similarly
+to with emacs, e.g.
+
+     make test TEST_CC=gcc TEST_CFLAGS="-g -O2"
+
+Parallel Execution
+------------------
+If either the moreutils or GNU "parallel" utility is available all
+tests will be run in parallel.  If the NOTMUCH_TEST_SERIALIZE variable
+is non-null all tests will be executed sequentially.
+
+Quiet Execution
+---------------
+Normally, when new script starts and when test PASSes you get a message
+printed on screen. This printing can be disabled by setting the
+NOTMUCH_TEST_QUIET variable to a non-null value. Message on test
+failures and skips are still printed.
+

 Skipping Tests
 --------------
 If, for any reason, you need to skip one or more tests, you can do so
 Skipping Tests
 --------------
 If, for any reason, you need to skip one or more tests, you can do so


@@ -75,16 +130,40 @@ Note that some tests in the existing test suite rely on previous test
 items, so you cannot arbitrarily skip any test and expect the
 remaining tests to be unaffected.
 
 items, so you cannot arbitrarily skip any test and expect the
 remaining tests to be unaffected.
 

+Currently we do not consider skipped tests as build failures. For
+maximum robustness, when setting up automated build processes, you
+should explicitly skip tests, rather than relying on notmuch's
+detection of missing prerequisites. In the future we may treat tests
+unable to run because of missing prerequisites, but not explicitly
+skipped by the user, as failures.
+
+Testing installed notmuch
+-------------------------
+
+Systems integrators (e.g. Linux distros) may wish to test an installed
+version of notmuch.  This can be done be running
+
+     $ NOTMUCH_TEST_INSTALLED=1 ./test/notmuch-test
+
+In this scenario the test suite does not assume a built tree, and in
+particular cannot rely on the output of 'configure'. You may want to
+set certain feature environment variables ('NOTMUCH_HAVE_*') directly
+if you know those apply to your installed notmuch). Consider also
+setting TERM=dumb if the value of TERM cannot be used (e.g. in a
+chroot with missing terminfo). Note that having a built tree may cause
+surprising/broken results for NOTMUCH_TEST_INSTALLED, so consider
+cleaning first.
+

 Writing Tests
 -------------
 Writing Tests
 -------------

-The test script is written as a shell script.  It should start with
-the standard "#!/usr/bin/env bash" with copyright notices, and an
-assignment to variable 'test_description', like this:
+The test script is written as a shell script. It is to be named as
+Tddd-testname.sh where 'ddd' is three digits and 'testname' the "bare"
+name of your test. Tests will be run in order the 'ddd' part determines.
+
+The test script should start with the standard "#!/usr/bin/env bash"
+and an assignment to variable 'test_description', like this:

 
        #!/usr/bin/env bash
 
        #!/usr/bin/env bash

-       #
-       # Copyright (c) 2005 Junio C Hamano
-       #

 
        test_description='xxx test (option --frotz)
 
 
        test_description='xxx test (option --frotz)
 


@@ -96,7 +175,7 @@ Source 'test-lib.sh'
 After assigning test_description, the test script should source
 test-lib.sh like this:
 
 After assigning test_description, the test script should source
 test-lib.sh like this:
 

-       . ./test-lib.sh
+       . ./test-lib.sh || exit 1

 
 This test harness library does the following things:
 
 
 This test harness library does the following things:
 


@@ -126,25 +205,29 @@ Test harness library
 There are a handful helper functions defined in the test harness
 library for your script to use.
 
 There are a handful helper functions defined in the test harness
 library for your script to use.
 

- test_expect_success <message> <script>
+ test_begin_subtest <message>
+
+   Set the test description message for a subsequent test_expect_*
+   invocation (see below).
+
+ test_expect_success <script>

 
 

-   This takes two strings as parameter, and evaluates the
+   This takes a string as parameter, and evaluates the

    <script>.  If it yields success, test is considered
    <script>.  If it yields success, test is considered

-   successful.  <message> should state what it is testing.
+   successful.

 
 

- test_expect_failure <message> <script>
+ test_expect_code <code> <script>

 
 

-   This is NOT the opposite of test_expect_success, but is used
-   to mark a test that demonstrates a known breakage.  Unlike
-   the usual test_expect_success tests, which say "ok" on
-   success and "FAIL" on failure, this will say "FIXED" on
-   success and "still broken" on failure.  Failures from these
-   tests won't cause -i (immediate) to stop.
+   This takes two strings as parameter, and evaluates the <script>.
+   If it yields <code> exit status, test is considered successful.

 
 

- test_begin_subtest <message>
+ test_subtest_known_broken

 
 

-   Set the test description message for a subsequent test_expect_equal
-   invocation (see below).
+   Mark the current test as broken.  Such tests are expected to fail.
+   Unlike the normal tests, which say "PASS" on success and "FAIL" on
+   failure, these will say "FIXED" on success and "BROKEN" on failure.
+   Failures from these tests won't cause -i (immediate) to stop.  A
+   test must call this before any test_expect_* function.

 
  test_expect_equal <output> <expected>
 
 
  test_expect_equal <output> <expected>
 


@@ -156,19 +239,19 @@ library for your script to use.
    will generate a failure and print the difference of the two
    strings.
 
    will generate a failure and print the difference of the two
    strings.
 

- test_expect_equal_file <output> <expected>
+ test_expect_equal_file <file1> <file2>

 
 

-   Identical to test_exepect_equal, except that <output> and
-   <expected> are files instead of strings.  This is a much more
-   robust method to compare formatted textual information, since it
-   also notices whitespace and closing newline differences.
+   Identical to test_expect_equal, except that <file1> and <file2>
+   are files instead of strings.  This is a much more robust method to
+   compare formatted textual information, since it also notices
+   whitespace and closing newline differences.

 
 

- test_expect_equal_failure <output> <expected>
+ test_expect_equal_json <output> <expected>

 
 

-   This works similar to test_expect_equal (see above) but is used to
-   mark a test that demonstrates a known breakage, (that is, the
-   author of the test expects "output" and "expected" to differ until
-   the breakage is fixed). See test_expect_failure for details.
+   Identical to test_expect_equal, except that the two strings are
+   treated as JSON and canonicalized before equality testing.  This is
+   useful to abstract away from whitespace differences in the expected
+   output and that generated by running a notmuch command.

 
  test_debug <script>
 
 
  test_debug <script>
 


@@ -181,9 +264,21 @@ library for your script to use.
 
    This function executes the provided emacs lisp script within
    emacs. The script can be a sequence of emacs lisp expressions,
 
    This function executes the provided emacs lisp script within
    emacs. The script can be a sequence of emacs lisp expressions,

-   (that is, they will be evaluated within a progn form). The lisp
-   expressions can call `message' to generate output on stdout to be
-   examined by the calling test script.
+   (that is, they will be evaluated within a progn form). Emacs
+   stdout and stderr is not available, the common way to get output
+   is to save it to a file. There are some auxiliary functions
+   useful in emacs tests provided in test-lib.el. Do not use `setq'
+   for setting variables in Emacs tests because it affects other
+   tests that may run in the same Emacs instance.  Use `let' instead
+   so the scope of the changed variables is limited to a single test.
+
+ test_emacs_expect_t <emacs-lisp-expressions>
+
+  This function executes the provided emacs lisp script within
+  emacs in a manner similar to 'test_emacs'. The expressions should
+  return the value `t' to indicate that the test has passed. If the
+  test does not return `t' then it is considered failed and all data
+  returned by the test is reported to the tester.

 
  test_done
 
 
  test_done
 


@@ -191,8 +286,8 @@ library for your script to use.
    is to summarize successes and failures in the test script and
    exit with an appropriate error code.
 
    is to summarize successes and failures in the test script and
    exit with an appropriate error code.
 

-There are also a number of mail-specific functions which are useful in
-writing tests:
+There are also a number of notmuch-specific auxiliary functions and
+variables which are useful in writing tests:

 
   generate_message
 
 
   generate_message
 


@@ -216,3 +311,28 @@ writing tests:
     will initialize the mail database to a known state of 50 sample
     messages, (culled from the early history of the notmuch mailing
     list).
     will initialize the mail database to a known state of 50 sample
     messages, (culled from the early history of the notmuch mailing
     list).

+
+  notmuch_counter_reset
+  $notmuch_counter_command
+  notmuch_counter_value
+
+    These allow to count how many times notmuch binary is called.
+    notmuch_counter_reset() function generates a script that counts
+    how many times it is called and resets the counter to zero.  The
+    function sets $notmuch_counter_command variable to the path to the
+    generated script that should be called instead of notmuch to do
+    the counting.  The notmuch_counter_value() function prints the
+    current counter value.
+
+There are also functions which remove various environment-dependent
+values from notmuch output; these are useful to ensure that test
+results remain consistent across different machines.
+
+ notmuch_search_sanitize
+ notmuch_show_sanitize
+ notmuch_show_sanitize_all
+ notmuch_json_show_sanitize
+
+   All these functions should receive the text to be sanitized as the
+   input of a pipe, e.g.
+   output=`notmuch search "..." | notmuch_search_sanitize`