test: Add `test_emacs_expect_t'.

[notmuch] / test / README

diff --git a/test/README b/test/README
index 5861cc498027af53cbf3bcde2aec61e274bd4e13..43656a35baf9f9bc5e5d41aca2b65c8e4e763450 100644 (file)
--- a/test/README
+++ b/test/README
@@ -6,6 +6,19 @@ When fixing bugs or enhancing notmuch, you are strongly encouraged to
 add tests in this directory to cover what you are trying to fix or
 enhance.
 
 add tests in this directory to cover what you are trying to fix or
 enhance.
 

+Prerequisites
+-------------
+Some tests require external dependencies to run. Without them, they
+will be skipped, or (rarely) marked failed. Please install these, so
+that you know if you break anything.
+
+  - dtach(1)
+  - emacs(1)
+  - emacsclient(1)
+  - gdb(1)
+  - gpg(1)
+  - python(1)
+

 Running Tests
 -------------
 The easiest way to run tests is to say "make test", (or simply run the
 Running Tests
 -------------
 The easiest way to run tests is to say "make test", (or simply run the


@@ -13,15 +26,11 @@ notmuch-test script). Either command will run all available tests.
 
 Alternately, you can run a specific subset of tests by simply invoking
 one of the executable scripts in this directory, (such as ./search,
 
 Alternately, you can run a specific subset of tests by simply invoking
 one of the executable scripts in this directory, (such as ./search,

-./reply, etc.)
+./reply, etc). Note that you will probably want "make test-binaries"
+before running individual tests.

 
 The following command-line options are available when running tests:
 
 
 The following command-line options are available when running tests:
 

---verbose::
-       This makes the test more verbose.  Specifically, the
-       command being run and their output if any are also
-       output.
-

 --debug::
        This may help the person who is developing a new test.
        It causes the command defined with test_debug to run.
 --debug::
        This may help the person who is developing a new test.
        It causes the command defined with test_debug to run.


@@ -46,11 +55,27 @@ The following command-line options are available when running tests:
        As the names depend on the tests' file names, it is safe to
        run the tests with this option in parallel.
 
        As the names depend on the tests' file names, it is safe to
        run the tests with this option in parallel.
 

+--root=<dir>::
+       This runs the testsuites specified under a seperate directory.
+       However, caution is advised, as not all tests are maintained
+       with this relocation in mind, so some tests may behave
+       differently.
+
+       Pointing this argument at a tmpfs filesystem can improve the
+       speed of the test suite for some users.
+

 When invoking the test suite via "make test" any of the above options
 can be specified as follows:
 
        make test OPTIONS="--verbose"
 
 When invoking the test suite via "make test" any of the above options
 can be specified as follows:
 
        make test OPTIONS="--verbose"
 

+You can choose an emacs binary to run the tests in one of the
+following ways.
+
+       TEST_EMACS=my-special-emacs make test
+       TEST_EMACS=my-special-emacs ./emacs
+       make test TEST_EMACS=my-special-emacs
+

 Skipping Tests
 --------------
 If, for any reason, you need to skip one or more tests, you can do so
 Skipping Tests
 --------------
 If, for any reason, you need to skip one or more tests, you can do so


@@ -73,11 +98,11 @@ remaining tests to be unaffected.
 
 Writing Tests
 -------------
 
 Writing Tests
 -------------

-The test script is written as a shell script.  It should start
-with the standard "#!/bin/bash" with copyright notices, and an
+The test script is written as a shell script.  It should start with
+the standard "#!/usr/bin/env bash" with copyright notices, and an

 assignment to variable 'test_description', like this:
 
 assignment to variable 'test_description', like this:
 

-       #!/bin/bash
+       #!/usr/bin/env bash

        #
        # Copyright (c) 2005 Junio C Hamano
        #
        #
        # Copyright (c) 2005 Junio C Hamano
        #


@@ -99,10 +124,12 @@ This test harness library does the following things:
  - If the script is invoked with command line argument --help
    (or -h), it shows the test_description and exits.
 
  - If the script is invoked with command line argument --help
    (or -h), it shows the test_description and exits.
 

- - Creates a temporary directory with default notmuch-config and empty
-   mail store. This directory is 'test/tmp.<test-basename>'. The path
-   to notmuch-config is exported in NOTMUCH_CONFIG environment
-   variable and mail store path is stored in MAIL_DIR variable.
+ - Creates a temporary directory with default notmuch-config and a
+   mail store with a corpus of mail, (initially, 50 early messages
+   sent to the notmuch list). This directory is
+   test/tmp.<test-basename>. The path to notmuch-config is exported in
+   NOTMUCH_CONFIG environment variable and mail store path is stored
+   in MAIL_DIR variable.

 
  - Defines standard test helper functions for your scripts to
    use.  These functions are designed to make all scripts behave
 
  - Defines standard test helper functions for your scripts to
    use.  These functions are designed to make all scripts behave


@@ -120,27 +147,26 @@ Test harness library
 There are a handful helper functions defined in the test harness
 library for your script to use.
 
 There are a handful helper functions defined in the test harness
 library for your script to use.
 

- - test_expect_success <message> <script>
+ test_expect_success <message> <script>

 
    This takes two strings as parameter, and evaluates the
    <script>.  If it yields success, test is considered
    successful.  <message> should state what it is testing.
 
 
    This takes two strings as parameter, and evaluates the
    <script>.  If it yields success, test is considered
    successful.  <message> should state what it is testing.
 

- - test_expect_failure <message> <script>
-
-   This is NOT the opposite of test_expect_success, but is used
-   to mark a test that demonstrates a known breakage.  Unlike
-   the usual test_expect_success tests, which say "ok" on
-   success and "FAIL" on failure, this will say "FIXED" on
-   success and "still broken" on failure.  Failures from these
-   tests won't cause -i (immediate) to stop.
-
- - test_begin_subtest <message>
+ test_begin_subtest <message>

 
    Set the test description message for a subsequent test_expect_equal
    invocation (see below).
 
 
    Set the test description message for a subsequent test_expect_equal
    invocation (see below).
 

- - test_expect_equal <output> <expected>
+ test_subtest_known_broken
+
+   Mark the current test as broken.  Such tests are expected to fail.
+   Unlike the normal tests, which say "PASS" on success and "FAIL" on
+   failure, these will say "FIXED" on success and "BROKEN" on failure.
+   Failures from these tests won't cause -i (immediate) to stop.  A
+   test must call this before any test_expect_* function.
+
+ test_expect_equal <output> <expected>

 
    This is an often-used convenience function built on top of
    test_expect_success. It uses the message from the last
 
    This is an often-used convenience function built on top of
    test_expect_success. It uses the message from the last


@@ -150,16 +176,80 @@ library for your script to use.
    will generate a failure and print the difference of the two
    strings.
 
    will generate a failure and print the difference of the two
    strings.
 

- - test_debug <script>
+ test_expect_equal_file <output> <expected>
+
+   Identical to test_exepect_equal, except that <output> and
+   <expected> are files instead of strings.  This is a much more
+   robust method to compare formatted textual information, since it
+   also notices whitespace and closing newline differences.
+
+ test_debug <script>

 
    This takes a single argument, <script>, and evaluates it only
    when the test script is started with --debug command line
    argument.  This is primarily meant for use during the
    development of a new test script.
 
 
    This takes a single argument, <script>, and evaluates it only
    when the test script is started with --debug command line
    argument.  This is primarily meant for use during the
    development of a new test script.
 

- - test_done
+ test_emacs <emacs-lisp-expressions>
+
+   This function executes the provided emacs lisp script within
+   emacs. The script can be a sequence of emacs lisp expressions,
+   (that is, they will be evaluated within a progn form). Emacs
+   stdout and stderr is not available, the common way to get output
+   is to save it to a file. There are some auxiliary functions
+   useful in emacs tests provided in test-lib.el. Do not use `setq'
+   for setting variables in Emacs tests because it affects other
+   tests that may run in the same Emacs instance.  Use `let' instead
+   so the scope of the changed variables is limited to a single test.
+
+ test_emacs_expect_t <emacs-lisp-expressions>
+
+  This function executes the provided emacs lisp script within
+  emacs in a manner similar to 'test_emacs'. The expressions should
+  return the value `t' to indicate that the test has passed. If the
+  test does not return `t' then it is considered failed and all data
+  returned by the test is reported to the tester.
+
+ test_done

 
    Your test script must have test_done at the end.  Its purpose
    is to summarize successes and failures in the test script and
    exit with an appropriate error code.
 
    Your test script must have test_done at the end.  Its purpose
    is to summarize successes and failures in the test script and
    exit with an appropriate error code.

-k
\ No newline at end of file
+
+There are also a number of notmuch-specific auxiliary functions and
+variables which are useful in writing tests:
+
+  generate_message
+
+    Generates a message with an optional template. Most tests will
+    actually prefer to call add_message. See below.
+
+  add_message
+
+    Generate a message and add it to the database (by calling "notmuch
+    new"). It is sufficient to simply call add_message with no
+    arguments if you don't care about the content of the message. If
+    more control is needed, arguments can be provide to specify many
+    different header values for the new message. See the documentation
+    within test-lib.sh or refer to many example calls within existing
+    tests.
+
+  add_email_corpus
+
+    This function should be called at the beginning of a test file
+    when a test needs to operate on a non-empty body of messages. It
+    will initialize the mail database to a known state of 50 sample
+    messages, (culled from the early history of the notmuch mailing
+    list).
+
+  notmuch_counter_reset
+  $notmuch_counter_command
+  notmuch_counter_value
+
+    These allow to count how many times notmuch binary is called.
+    notmuch_counter_reset() function generates a script that counts
+    how many times it is called and resets the counter to zero.  The
+    function sets $notmuch_counter_command variable to the path to the
+    generated script that should be called instead of notmuch to do
+    the counting.  The notmuch_counter_value() function prints the
+    current counter value.