Merge tag 'debian/0.17-4'

[notmuch] / test / README

diff --git a/test/README b/test/README
index 2e757e0eeca8d1c7641848bbb744ee681008fb9c..79a9b1b2f9a15aba7466e4e5b1b308663dd1296c 100644 (file)
--- a/test/README
+++ b/test/README
@@ -6,6 +6,19 @@ When fixing bugs or enhancing notmuch, you are strongly encouraged to
 add tests in this directory to cover what you are trying to fix or
 enhance.
 
 add tests in this directory to cover what you are trying to fix or
 enhance.
 

+Prerequisites
+-------------
+Some tests require external dependencies to run. Without them, they
+will be skipped, or (rarely) marked failed. Please install these, so
+that you know if you break anything.
+
+  - dtach(1)
+  - emacs(1)
+  - emacsclient(1)
+  - gdb(1)
+  - gpg(1)
+  - python(1)
+

 Running Tests
 -------------
 The easiest way to run tests is to say "make test", (or simply run the
 Running Tests
 -------------
 The easiest way to run tests is to say "make test", (or simply run the


@@ -43,7 +56,7 @@ The following command-line options are available when running tests:
        run the tests with this option in parallel.
 
 --root=<dir>::
        run the tests with this option in parallel.
 
 --root=<dir>::

-       This runs the testsuites specified under a seperate directory.
+       This runs the testsuites specified under a separate directory.

        However, caution is advised, as not all tests are maintained
        with this relocation in mind, so some tests may behave
        differently.
        However, caution is advised, as not all tests are maintained
        with this relocation in mind, so some tests may behave
        differently.


@@ -56,6 +69,21 @@ can be specified as follows:
 
        make test OPTIONS="--verbose"
 
 
        make test OPTIONS="--verbose"
 

+You can choose an emacs binary (and corresponding emacsclient) to run
+the tests in one of the following ways.
+
+       TEST_EMACS=my-special-emacs TEST_EMACSCLIENT=my-emacsclient make test
+       TEST_EMACS=my-special-emacs TEST_EMACSCLIENT=my-emacsclient ./emacs
+       make test TEST_EMACS=my-special-emacs TEST_EMACSCLIENT=my-emacsclient
+
+Quiet Execution
+---------------
+
+Normally, when new script starts and when test PASSes you get a message
+printed on screen. This printing can be disabled by setting the
+NOTMUCH_TEST_QUIET variable to a non-null value. Message on test
+failures and skips are still printed.
+

 Skipping Tests
 --------------
 If, for any reason, you need to skip one or more tests, you can do so
 Skipping Tests
 --------------
 If, for any reason, you need to skip one or more tests, you can do so


@@ -156,12 +184,19 @@ library for your script to use.
    will generate a failure and print the difference of the two
    strings.
 
    will generate a failure and print the difference of the two
    strings.
 

- test_expect_equal_file <output> <expected>
+ test_expect_equal_file <file1> <file2>
+
+   Identical to test_expect_equal, except that <file1> and <file2>
+   are files instead of strings.  This is a much more robust method to
+   compare formatted textual information, since it also notices
+   whitespace and closing newline differences.
+
+ test_expect_equal_json <output> <expected>

 
 

-   Identical to test_exepect_equal, except that <output> and
-   <expected> are files instead of strings.  This is a much more
-   robust method to compare formatted textual information, since it
-   also notices whitespace and closing newline differences.
+   Identical to test_expect_equal, except that the two strings are
+   treated as JSON and canonicalized before equality testing.  This is
+   useful to abstract away from whitespace differences in the expected
+   output and that generated by running a notmuch command.

 
  test_debug <script>
 
 
  test_debug <script>
 


@@ -182,14 +217,22 @@ library for your script to use.
    tests that may run in the same Emacs instance.  Use `let' instead
    so the scope of the changed variables is limited to a single test.
 
    tests that may run in the same Emacs instance.  Use `let' instead
    so the scope of the changed variables is limited to a single test.
 

+ test_emacs_expect_t <emacs-lisp-expressions>
+
+  This function executes the provided emacs lisp script within
+  emacs in a manner similar to 'test_emacs'. The expressions should
+  return the value `t' to indicate that the test has passed. If the
+  test does not return `t' then it is considered failed and all data
+  returned by the test is reported to the tester.
+

  test_done
 
    Your test script must have test_done at the end.  Its purpose
    is to summarize successes and failures in the test script and
    exit with an appropriate error code.
 
  test_done
 
    Your test script must have test_done at the end.  Its purpose
    is to summarize successes and failures in the test script and
    exit with an appropriate error code.
 

-There are also a number of mail-specific functions which are useful in
-writing tests:
+There are also a number of notmuch-specific auxiliary functions and
+variables which are useful in writing tests:

 
   generate_message
 
 
   generate_message
 


@@ -213,3 +256,28 @@ writing tests:
     will initialize the mail database to a known state of 50 sample
     messages, (culled from the early history of the notmuch mailing
     list).
     will initialize the mail database to a known state of 50 sample
     messages, (culled from the early history of the notmuch mailing
     list).

+
+  notmuch_counter_reset
+  $notmuch_counter_command
+  notmuch_counter_value
+
+    These allow to count how many times notmuch binary is called.
+    notmuch_counter_reset() function generates a script that counts
+    how many times it is called and resets the counter to zero.  The
+    function sets $notmuch_counter_command variable to the path to the
+    generated script that should be called instead of notmuch to do
+    the counting.  The notmuch_counter_value() function prints the
+    current counter value.
+
+There are also functions which remove various environment-dependent
+values from notmuch output; these are useful to ensure that test
+results remain consistent across different machines.
+
+ notmuch_search_sanitize
+ notmuch_show_sanitize
+ notmuch_show_sanitize_all
+ notmuch_json_show_sanitize
+
+   All these functions should receive the text to be sanitized as the
+   input of a pipe, e.g.
+   output=`notmuch search "..." | notmuch_search_sanitize`