test: improve known broken tests support

[notmuch] / test / README

diff --git a/test/README b/test/README
index ebaa3cfd2c732dde689ee6055883dabca54ab598..0b547480e5497e80b05c3001265ead2cb4f7dc9a 100644 (file)
--- a/test/README
+++ b/test/README
@@ -41,6 +41,15 @@ The following command-line options are available when running tests:
        As the names depend on the tests' file names, it is safe to
        run the tests with this option in parallel.
 
        As the names depend on the tests' file names, it is safe to
        run the tests with this option in parallel.
 

+--root=<dir>::
+       This runs the testsuites specified under a seperate directory.
+       However, caution is advised, as not all tests are maintained
+       with this relocation in mind, so some tests may behave
+       differently.
+
+       Pointing this argument at a tmpfs filesystem can improve the
+       speed of the test suite for some users.
+

 When invoking the test suite via "make test" any of the above options
 can be specified as follows:
 
 When invoking the test suite via "make test" any of the above options
 can be specified as follows:
 


@@ -68,11 +77,11 @@ remaining tests to be unaffected.
 
 Writing Tests
 -------------
 
 Writing Tests
 -------------

-The test script is written as a shell script.  It should start
-with the standard "#!/bin/bash" with copyright notices, and an
+The test script is written as a shell script.  It should start with
+the standard "#!/usr/bin/env bash" with copyright notices, and an

 assignment to variable 'test_description', like this:
 
 assignment to variable 'test_description', like this:
 

-       #!/bin/bash
+       #!/usr/bin/env bash

        #
        # Copyright (c) 2005 Junio C Hamano
        #
        #
        # Copyright (c) 2005 Junio C Hamano
        #


@@ -123,20 +132,19 @@ library for your script to use.
    <script>.  If it yields success, test is considered
    successful.  <message> should state what it is testing.
 
    <script>.  If it yields success, test is considered
    successful.  <message> should state what it is testing.
 

- test_expect_failure <message> <script>
-
-   This is NOT the opposite of test_expect_success, but is used
-   to mark a test that demonstrates a known breakage.  Unlike
-   the usual test_expect_success tests, which say "ok" on
-   success and "FAIL" on failure, this will say "FIXED" on
-   success and "still broken" on failure.  Failures from these
-   tests won't cause -i (immediate) to stop.
-

  test_begin_subtest <message>
 
    Set the test description message for a subsequent test_expect_equal
    invocation (see below).
 
  test_begin_subtest <message>
 
    Set the test description message for a subsequent test_expect_equal
    invocation (see below).
 

+ test_subtest_known_broken
+
+   Mark the current test as broken.  Such tests are expected to fail.
+   Unlike the normal tests, which say "PASS" on success and "FAIL" on
+   failure, these will say "FIXED" on success and "BROKEN" on failure.
+   Failures from these tests won't cause -i (immediate) to stop.  A
+   test must call this before any test_expect_* function.
+

  test_expect_equal <output> <expected>
 
    This is an often-used convenience function built on top of
  test_expect_equal <output> <expected>
 
    This is an often-used convenience function built on top of


@@ -147,11 +155,18 @@ library for your script to use.
    will generate a failure and print the difference of the two
    strings.
 
    will generate a failure and print the difference of the two
    strings.
 

+ test_expect_equal_file <output> <expected>
+
+   Identical to test_exepect_equal, except that <output> and
+   <expected> are files instead of strings.  This is a much more
+   robust method to compare formatted textual information, since it
+   also notices whitespace and closing newline differences.
+

  test_expect_equal_failure <output> <expected>
 
    This works similar to test_expect_equal (see above) but is used to
    mark a test that demonstrates a known breakage, (that is, the
  test_expect_equal_failure <output> <expected>
 
    This works similar to test_expect_equal (see above) but is used to
    mark a test that demonstrates a known breakage, (that is, the

-   author of the test expectes "output" and "expected" to differ until
+   author of the test expects "output" and "expected" to differ until

    the breakage is fixed). See test_expect_failure for details.
 
  test_debug <script>
    the breakage is fixed). See test_expect_failure for details.
 
  test_debug <script>


@@ -165,9 +180,13 @@ library for your script to use.
 
    This function executes the provided emacs lisp script within
    emacs. The script can be a sequence of emacs lisp expressions,
 
    This function executes the provided emacs lisp script within
    emacs. The script can be a sequence of emacs lisp expressions,

-   (that is, they will be evaluated within a progn form). The lisp
-   expressions can call `message' to generate output on stdout to be
-   examined by the calling test script.
+   (that is, they will be evaluated within a progn form). Emacs
+   stdout and stderr is not available, the common way to get output
+   is to save it to a file. There are some auxiliary functions
+   useful in emacs tests provided in test-lib.el. Do not use `setq'
+   for setting variables in Emacs tests because it affects other
+   tests that may run in the same Emacs instance.  Use `let' instead
+   so the scope of the changed variables is limited to a single test.

 
  test_done
 
 
  test_done
 


@@ -181,7 +200,7 @@ writing tests:
   generate_message
 
     Generates a message with an optional template. Most tests will
   generate_message
 
     Generates a message with an optional template. Most tests will

-    actually prefere to call add_message. See below.
+    actually prefer to call add_message. See below.

 
   add_message
 
 
   add_message
 


@@ -197,6 +216,6 @@ writing tests:
 
     This function should be called at the beginning of a test file
     when a test needs to operate on a non-empty body of messages. It
 
     This function should be called at the beginning of a test file
     when a test needs to operate on a non-empty body of messages. It

-    will intialize the mail database to a known state of 50 sample
+    will initialize the mail database to a known state of 50 sample

     messages, (culled from the early history of the notmuch mailing
     list).
     messages, (culled from the early history of the notmuch mailing
     list).