add tests in this directory to cover what you are trying to fix or
enhance.
+Prerequisites
+-------------
+Some tests require external dependencies to run. Without them, they
+will be skipped, or (rarely) marked failed. Please install these, so
+that you know if you break anything.
+
+ - dtach(1)
+ - emacs(1)
+ - emacsclient(1)
+ - gdb(1)
+ - gpg(1)
+ - python(1)
+
Running Tests
-------------
The easiest way to run tests is to say "make test", (or simply run the
run the tests with this option in parallel.
--root=<dir>::
- This runs the testsuites specified under a seperate directory.
+ This runs the testsuites specified under a separate directory.
However, caution is advised, as not all tests are maintained
with this relocation in mind, so some tests may behave
differently.
Pointing this argument at a tmpfs filesystem can improve the
speed of the test suite for some users.
+Certain tests require precomputed databases to complete. You can fetch these
+databases with
+
+ make download-test-databases
+
+If you do not download the test databases, the relevant tests will be
+skipped.
+
When invoking the test suite via "make test" any of the above options
can be specified as follows:
make test OPTIONS="--verbose"
+You can choose an emacs binary (and corresponding emacsclient) to run
+the tests in one of the following ways.
+
+ TEST_EMACS=my-special-emacs TEST_EMACSCLIENT=my-emacsclient make test
+ TEST_EMACS=my-special-emacs TEST_EMACSCLIENT=my-emacsclient ./emacs
+ make test TEST_EMACS=my-special-emacs TEST_EMACSCLIENT=my-emacsclient
+
+Some tests may require a c compiler. You can choose the name and flags similarly
+to with emacs, e.g.
+
+ make test TEST_CC=gcc TEST_CFLAGS="-g -O2"
+
+Quiet Execution
+---------------
+
+Normally, when new script starts and when test PASSes you get a message
+printed on screen. This printing can be disabled by setting the
+NOTMUCH_TEST_QUIET variable to a non-null value. Message on test
+failures and skips are still printed.
+
Skipping Tests
--------------
If, for any reason, you need to skip one or more tests, you can do so
will generate a failure and print the difference of the two
strings.
- test_expect_equal_file <output> <expected>
+ test_expect_equal_file <file1> <file2>
+
+ Identical to test_expect_equal, except that <file1> and <file2>
+ are files instead of strings. This is a much more robust method to
+ compare formatted textual information, since it also notices
+ whitespace and closing newline differences.
- Identical to test_exepect_equal, except that <output> and
- <expected> are files instead of strings. This is a much more
- robust method to compare formatted textual information, since it
- also notices whitespace and closing newline differences.
+ test_expect_equal_json <output> <expected>
+
+ Identical to test_expect_equal, except that the two strings are
+ treated as JSON and canonicalized before equality testing. This is
+ useful to abstract away from whitespace differences in the expected
+ output and that generated by running a notmuch command.
test_debug <script>
tests that may run in the same Emacs instance. Use `let' instead
so the scope of the changed variables is limited to a single test.
+ test_emacs_expect_t <emacs-lisp-expressions>
+
+ This function executes the provided emacs lisp script within
+ emacs in a manner similar to 'test_emacs'. The expressions should
+ return the value `t' to indicate that the test has passed. If the
+ test does not return `t' then it is considered failed and all data
+ returned by the test is reported to the tester.
+
test_done
Your test script must have test_done at the end. Its purpose
is to summarize successes and failures in the test script and
exit with an appropriate error code.
-There are also a number of mail-specific functions which are useful in
-writing tests:
+There are also a number of notmuch-specific auxiliary functions and
+variables which are useful in writing tests:
generate_message
will initialize the mail database to a known state of 50 sample
messages, (culled from the early history of the notmuch mailing
list).
+
+ notmuch_counter_reset
+ $notmuch_counter_command
+ notmuch_counter_value
+
+ These allow to count how many times notmuch binary is called.
+ notmuch_counter_reset() function generates a script that counts
+ how many times it is called and resets the counter to zero. The
+ function sets $notmuch_counter_command variable to the path to the
+ generated script that should be called instead of notmuch to do
+ the counting. The notmuch_counter_value() function prints the
+ current counter value.
+
+There are also functions which remove various environment-dependent
+values from notmuch output; these are useful to ensure that test
+results remain consistent across different machines.
+
+ notmuch_search_sanitize
+ notmuch_show_sanitize
+ notmuch_show_sanitize_all
+ notmuch_json_show_sanitize
+
+ All these functions should receive the text to be sanitized as the
+ input of a pipe, e.g.
+ output=`notmuch search "..." | notmuch_search_sanitize`