-Core GIT Tests
-==============
+Notmuch test suite
+==================
+This directory contains the test suite for notmuch.
-This directory holds many test scripts for core GIT tools. The
-first part of this short document describes how to run the tests
-and read their output.
-
-When fixing the tools or adding enhancements, you are strongly
-encouraged to add tests in this directory to cover what you are
-trying to fix or enhance. The later part of this short document
-describes how your test scripts should be organized.
+When fixing bugs or enhancing notmuch, you are strongly encouraged to
+add tests in this directory to cover what you are trying to fix or
+enhance.
+Prerequisites
+-------------
+The test system itself requires:
+
+ - bash(1) version 4.0 or newer
+
+Without bash 4.0+ the tests just refuse to run.
+
+Some tests require external dependencies to run. Without them, they
+will be skipped, or (rarely) marked failed. Please install these, so
+that you know if you break anything.
+
+ - GNU tar(1)
+ - dtach(1)
+ - emacs(1)
+ - emacsclient(1)
+ - gdb(1)
+ - gpg(1)
+ - python(1)
+ - xapian-metadata(1)
+
+If your system lacks these tools or have older, non-upgradable versions
+of these, please (possibly compile and) install these to some other
+path, for example /usr/local/bin or /opt/gnu/bin. Then prepend the
+chosen directory to your PATH before running the tests.
+
+e.g. env PATH=/opt/gnu/bin:$PATH make test
+
+For FreeBSD you need to install latest gdb from ports or packages and
+provide path to it in TEST_GDB environment variable before executing
+the tests, native FreeBSD gdb does not not work. If you install
+coreutils, which provides GNU versions of basic utils like 'date' and
+'base64' on FreeBSD, the test suite will use these instead of the
+native ones. This provides robustness against portability issues with
+these system tools. Most often the tests are written, reviewed and
+tested on Linux system so such portability issues arise from time to
+time.
Running Tests
-------------
+The easiest way to run tests is to say "make test", (or simply run the
+notmuch-test script). Either command will run all available tests.
+
+Alternately, you can run a specific subset of tests by simply invoking
+one of the executable scripts in this directory, (such as ./T*-search.sh,
+./T*-reply.sh, etc). Note that you will probably want "make test-binaries"
+before running individual tests.
-The easiest way to run tests is to say "make". This runs all
-the tests.
-
- *** t0000-basic.sh ***
- * ok 1: .git/objects should be empty after git-init in an empty repo.
- * ok 2: .git/objects should have 256 subdirectories.
- * ok 3: git-update-index without --add should fail adding.
- ...
- * ok 23: no diff after checkout and git-update-index --refresh.
- * passed all 23 test(s)
- *** t0100-environment-names.sh ***
- * ok 1: using old names should issue warnings.
- * ok 2: using old names but having new names should not issue warnings.
- ...
-
-Or you can run each test individually from command line, like
-this:
-
- $ sh ./t3001-ls-files-killed.sh
- * ok 1: git-update-index --add to add various paths.
- * ok 2: git-ls-files -k to show killed files.
- * ok 3: validate git-ls-files -k output.
- * passed all 3 test(s)
-
-You can pass --verbose (or -v), --debug (or -d), and --immediate
-(or -i) command line argument to the test, or by setting GIT_TEST_OPTS
-appropriately before running "make".
-
---verbose::
- This makes the test more verbose. Specifically, the
- command being run and their output if any are also
- output.
+The following command-line options are available when running tests:
--debug::
This may help the person who is developing a new test.
This causes the test to immediately exit upon the first
failed test.
---long-tests::
- This causes additional long-running tests to be run (where
- available), for more exhaustive testing.
-
--valgrind::
- Execute all Git binaries with valgrind and exit with status
+ Execute notmuch with valgrind and exit with status
126 on errors (just like regular tests, this will only stop
the test script when running under -i). Valgrind errors
go to stderr, so you might want to pass the -v option, too.
As the names depend on the tests' file names, it is safe to
run the tests with this option in parallel.
---with-dashes::
- By default tests are run without dashed forms of
- commands (like git-commit) in the PATH (it only uses
- wrappers from ../bin-wrappers). Use this option to include
- the build directory (..) in the PATH, which contains all
- the dashed forms of commands. This option is currently
- implied by other options like --valgrind and
- GIT_TEST_INSTALLED.
+When invoking the test suite via "make test" any of the above options
+can be specified as follows:
-You can also set the GIT_TEST_INSTALLED environment variable to
-the bindir of an existing git installation to test that installation.
-You still need to have built this git sandbox, from which various
-test-* support programs, templates, and perl libraries are used.
-If your installed git is incomplete, it will silently test parts of
-your built version instead.
+ make test OPTIONS="--verbose"
-When using GIT_TEST_INSTALLED, you can also set GIT_TEST_EXEC_PATH to
-override the location of the dashed-form subcommands (what
-GIT_EXEC_PATH would be used for during normal operation).
-GIT_TEST_EXEC_PATH defaults to `$GIT_TEST_INSTALLED/git --exec-path`.
+You can choose an emacs binary (and corresponding emacsclient) to run
+the tests in one of the following ways.
+ TEST_EMACS=my-emacs TEST_EMACSCLIENT=my-emacsclient make test
+ TEST_EMACS=my-emacs TEST_EMACSCLIENT=my-emacsclient ./T*-emacs.sh
+ make test TEST_EMACS=my-emacs TEST_EMACSCLIENT=my-emacsclient
-Skipping Tests
---------------
+Some tests may require a c compiler. You can choose the name and flags similarly
+to with emacs, e.g.
-In some environments, certain tests have no way of succeeding
-due to platform limitation, such as lack of 'unzip' program, or
-filesystem that do not allow arbitrary sequence of non-NUL bytes
-as pathnames.
+ make test TEST_CC=gcc TEST_CFLAGS="-g -O2"
-You should be able to say something like
-
- $ GIT_SKIP_TESTS=t9200.8 sh ./t9200-git-cvsexport-commit.sh
-
-and even:
-
- $ GIT_SKIP_TESTS='t[0-4]??? t91?? t9200.8' make
-
-to omit such tests. The value of the environment variable is a
-SP separated list of patterns that tells which tests to skip,
-and either can match the "t[0-9]{4}" part to skip the whole
-test, or t[0-9]{4} followed by ".$number" to say which
-particular test to skip.
+Parallel Execution
+------------------
+If either the moreutils or GNU "parallel" utility is available all
+tests will be run in parallel. If the NOTMUCH_TEST_SERIALIZE variable
+is non-null all tests will be executed sequentially.
-Note that some tests in the existing test suite rely on previous
-test item, so you cannot arbitrarily disable one and expect the
-remainder of test to check what the test originally was intended
-to check.
+Quiet Execution
+---------------
+Normally, when new script starts and when test PASSes you get a message
+printed on screen. This printing can be disabled by setting the
+NOTMUCH_TEST_QUIET variable to a non-null value. Message on test
+failures and skips are still printed.
+Skipping Tests
+--------------
+If, for any reason, you need to skip one or more tests, you can do so
+by setting the NOTMUCH_SKIP_TESTS variable to the name of one or more
+sections of tests.
-Naming Tests
-------------
+For example:
-The test files are named as:
+ $ NOTMUCH_SKIP_TESTS="search reply" make test
- tNNNN-commandname-details.sh
+Even more fine-grained skipping is possible by appending a test number
+(or glob pattern) after the section name. For example, the first
+search test and the second reply test could be skipped with:
-where N is a decimal digit.
+ $ NOTMUCH_SKIP_TESTS="search.1 reply.2" make test
-First digit tells the family:
+Note that some tests in the existing test suite rely on previous test
+items, so you cannot arbitrarily skip any test and expect the
+remaining tests to be unaffected.
- 0 - the absolute basics and global stuff
- 1 - the basic commands concerning database
- 2 - the basic commands concerning the working tree
- 3 - the other basic commands (e.g. ls-files)
- 4 - the diff commands
- 5 - the pull and exporting commands
- 6 - the revision tree commands (even e.g. merge-base)
- 7 - the porcelainish commands concerning the working tree
- 8 - the porcelainish commands concerning forensics
- 9 - the git tools
+Currently we do not consider skipped tests as build failures. For
+maximum robustness, when setting up automated build processes, you
+should explicitly skip tests, rather than relying on notmuch's
+detection of missing prerequisites. In the future we may treat tests
+unable to run because of missing prerequisites, but not explicitly
+skipped by the user, as failures.
-Second digit tells the particular command we are testing.
+Testing installed notmuch
+-------------------------
-Third digit (optionally) tells the particular switch or group of switches
-we are testing.
+Systems integrators (e.g. Linux distros) may wish to test an installed
+version of notmuch. This can be done be running
-If you create files under t/ directory (i.e. here) that is not
-the top-level test script, never name the file to match the above
-pattern. The Makefile here considers all such files as the
-top-level test script and tries to run all of them. A care is
-especially needed if you are creating a common test library
-file, similar to test-lib.sh, because such a library file may
-not be suitable for standalone execution.
+ $ NOTMUCH_TEST_INSTALLED=1 ./test/notmuch-test
+In this scenario the test suite does not assume a built tree, and in
+particular cannot rely on the output of 'configure'. You may want to
+set certain feature environment variables ('NOTMUCH_HAVE_*') directly
+if you know those apply to your installed notmuch). Consider also
+setting TERM=dumb if the value of TERM cannot be used (e.g. in a
+chroot with missing terminfo). Note that having a built tree may cause
+surprising/broken results for NOTMUCH_TEST_INSTALLED, so consider
+cleaning first.
Writing Tests
-------------
+The test script is written as a shell script. It is to be named as
+Tddd-testname.sh where 'ddd' is three digits and 'testname' the "bare"
+name of your test. Tests will be run in order the 'ddd' part determines.
-The test script is written as a shell script. It should start
-with the standard "#!/bin/bash" with copyright notices, and an
-assignment to variable 'test_description', like this:
+The test script should start with the standard "#!/usr/bin/env bash"
+and an assignment to variable 'test_description', like this:
- #!/bin/bash
- #
- # Copyright (c) 2005 Junio C Hamano
- #
+ #!/usr/bin/env bash
test_description='xxx test (option --frotz)
- This test registers the following structure in the cache
- and tries to run git-ls-files with option --frotz.'
-
+ This test exercises the "notmuch xxx" command when
+ given the option --frotz.'
Source 'test-lib.sh'
--------------------
-
After assigning test_description, the test script should source
test-lib.sh like this:
- . ./test-lib.sh
+ . ./test-lib.sh || exit 1
This test harness library does the following things:
- If the script is invoked with command line argument --help
(or -h), it shows the test_description and exits.
- - Creates a test directory with default notmuch-config and empty mail
- store. This directory is 't/trash directory.<test-basename>' (note
- the space) if you must know, but I do not think you care. The path
- to notmuch-config is exported in NOTMUCH_CONFIG environment
- variable and mail store path is stored in MAIL_DIR variable.
+ - Creates a temporary directory with default notmuch-config and a
+ mail store with a corpus of mail, (initially, 50 early messages
+ sent to the notmuch list). This directory is
+ test/tmp.<test-basename>. The path to notmuch-config is exported in
+ NOTMUCH_CONFIG environment variable and mail store path is stored
+ in MAIL_DIR variable.
- Defines standard test helper functions for your scripts to
use. These functions are designed to make all scripts behave
consistently when command line arguments --verbose (or -v),
--debug (or -d), and --immediate (or -i) is given.
-
End with test_done
------------------
-
Your script will be a sequence of tests, using helper functions
from the test harness library. At the end of the script, call
'test_done'.
-
Test harness library
--------------------
-
There are a handful helper functions defined in the test harness
library for your script to use.
- - test_expect_success <message> <script>
+ test_begin_subtest <message>
+
+ Set the test description message for a subsequent test_expect_*
+ invocation (see below).
+
+ test_expect_success <script>
- This takes two strings as parameter, and evaluates the
+ This takes a string as parameter, and evaluates the
<script>. If it yields success, test is considered
- successful. <message> should state what it is testing.
+ successful.
- Example:
+ test_expect_code <code> <script>
- test_expect_success \
- 'git-write-tree should be able to write an empty tree.' \
- 'tree=$(git-write-tree)'
+ This takes two strings as parameter, and evaluates the <script>.
+ If it yields <code> exit status, test is considered successful.
- - test_expect_failure <message> <script>
+ test_subtest_known_broken
- This is NOT the opposite of test_expect_success, but is used
- to mark a test that demonstrates a known breakage. Unlike
- the usual test_expect_success tests, which say "ok" on
- success and "FAIL" on failure, this will say "FIXED" on
- success and "still broken" on failure. Failures from these
- tests won't cause -i (immediate) to stop.
+ Mark the current test as broken. Such tests are expected to fail.
+ Unlike the normal tests, which say "PASS" on success and "FAIL" on
+ failure, these will say "FIXED" on success and "BROKEN" on failure.
+ Failures from these tests won't cause -i (immediate) to stop. A
+ test must call this before any test_expect_* function.
- - test_debug <script>
+ test_expect_equal <output> <expected>
+
+ This is an often-used convenience function built on top of
+ test_expect_success. It uses the message from the last
+ test_begin_subtest call, so call before calling
+ test_expect_equal. This function generates a successful test if
+ both the <output> and <expected> strings are identical. If not, it
+ will generate a failure and print the difference of the two
+ strings.
+
+ test_expect_equal_file <file1> <file2>
+
+ Identical to test_expect_equal, except that <file1> and <file2>
+ are files instead of strings. This is a much more robust method to
+ compare formatted textual information, since it also notices
+ whitespace and closing newline differences.
+
+ test_expect_equal_json <output> <expected>
+
+ Identical to test_expect_equal, except that the two strings are
+ treated as JSON and canonicalized before equality testing. This is
+ useful to abstract away from whitespace differences in the expected
+ output and that generated by running a notmuch command.
+
+ test_debug <script>
This takes a single argument, <script>, and evaluates it only
when the test script is started with --debug command line
argument. This is primarily meant for use during the
development of a new test script.
- - test_done
+ test_emacs <emacs-lisp-expressions>
+
+ This function executes the provided emacs lisp script within
+ emacs. The script can be a sequence of emacs lisp expressions,
+ (that is, they will be evaluated within a progn form). Emacs
+ stdout and stderr is not available, the common way to get output
+ is to save it to a file. There are some auxiliary functions
+ useful in emacs tests provided in test-lib.el. Do not use `setq'
+ for setting variables in Emacs tests because it affects other
+ tests that may run in the same Emacs instance. Use `let' instead
+ so the scope of the changed variables is limited to a single test.
+
+ test_emacs_expect_t <emacs-lisp-expressions>
+
+ This function executes the provided emacs lisp script within
+ emacs in a manner similar to 'test_emacs'. The expressions should
+ return the value `t' to indicate that the test has passed. If the
+ test does not return `t' then it is considered failed and all data
+ returned by the test is reported to the tester.
+
+ test_done
Your test script must have test_done at the end. Its purpose
is to summarize successes and failures in the test script and
exit with an appropriate error code.
- - test_tick
-
- Make commit and tag names consistent by setting the author and
- committer times to defined stated. Subsequent calls will
- advance the times by a fixed amount.
-
- - test_commit <message> [<filename> [<contents>]]
-
- Creates a commit with the given message, committing the given
- file with the given contents (default for both is to reuse the
- message string), and adds a tag (again reusing the message
- string as name). Calls test_tick to make the SHA-1s
- reproducible.
-
- - test_merge <message> <commit-or-tag>
-
- Merges the given rev using the given message. Like test_commit,
- creates a tag and calls test_tick before committing.
-
-Tips for Writing Tests
-----------------------
-
-As with any programming projects, existing programs are the best
-source of the information. However, do _not_ emulate
-t0000-basic.sh when writing your tests. The test is special in
-that it tries to validate the very core of GIT. For example, it
-knows that there will be 256 subdirectories under .git/objects/,
-and it knows that the object ID of an empty tree is a certain
-40-byte string. This is deliberately done so in t0000-basic.sh
-because the things the very basic core test tries to achieve is
-to serve as a basis for people who are changing the GIT internal
-drastically. For these people, after making certain changes,
-not seeing failures from the basic test _is_ a failure. And
-such drastic changes to the core GIT that even changes these
-otherwise supposedly stable object IDs should be accompanied by
-an update to t0000-basic.sh.
-
-However, other tests that simply rely on basic parts of the core
-GIT working properly should not have that level of intimate
-knowledge of the core GIT internals. If all the test scripts
-hardcoded the object IDs like t0000-basic.sh does, that defeats
-the purpose of t0000-basic.sh, which is to isolate that level of
-validation in one place. Your test also ends up needing
-updating when such a change to the internal happens, so do _not_
-do it and leave the low level of validation to t0000-basic.sh.
+There are also a number of notmuch-specific auxiliary functions and
+variables which are useful in writing tests:
+
+ generate_message
+
+ Generates a message with an optional template. Most tests will
+ actually prefer to call add_message. See below.
+
+ add_message
+
+ Generate a message and add it to the database (by calling "notmuch
+ new"). It is sufficient to simply call add_message with no
+ arguments if you don't care about the content of the message. If
+ more control is needed, arguments can be provide to specify many
+ different header values for the new message. See the documentation
+ within test-lib.sh or refer to many example calls within existing
+ tests.
+
+ add_email_corpus
+
+ This function should be called at the beginning of a test file
+ when a test needs to operate on a non-empty body of messages. It
+ will initialize the mail database to a known state of 50 sample
+ messages, (culled from the early history of the notmuch mailing
+ list).
+
+ notmuch_counter_reset
+ $notmuch_counter_command
+ notmuch_counter_value
+
+ These allow to count how many times notmuch binary is called.
+ notmuch_counter_reset() function generates a script that counts
+ how many times it is called and resets the counter to zero. The
+ function sets $notmuch_counter_command variable to the path to the
+ generated script that should be called instead of notmuch to do
+ the counting. The notmuch_counter_value() function prints the
+ current counter value.
+
+There are also functions which remove various environment-dependent
+values from notmuch output; these are useful to ensure that test
+results remain consistent across different machines.
+
+ notmuch_search_sanitize
+ notmuch_show_sanitize
+ notmuch_show_sanitize_all
+ notmuch_json_show_sanitize
+
+ All these functions should receive the text to be sanitized as the
+ input of a pipe, e.g.
+ output=`notmuch search "..." | notmuch_search_sanitize`
projects / notmuch / blobdiff