-Core GIT Tests
-==============
-
-This directory holds many test scripts for core GIT tools. The
-first part of this short document describes how to run the tests
-and read their output.
-
-When fixing the tools or adding enhancements, you are strongly
-encouraged to add tests in this directory to cover what you are
-trying to fix or enhance. The later part of this short document
-describes how your test scripts should be organized.
+Notmuch test suite
+==================
+This directory contains the test suite for notmuch.
+When fixing bugs or enhancing notmuch, you are strongly encouraged to
+add tests in this directory to cover what you are trying to fix or
+enhance.
Running Tests
-------------
+The easiest way to run tests is to say "make test", (or simply run the
+notmuch-test script). Either command will run all available tests.
+
+Alternately, you can run a specific subset of tests by simply invoking
+one of the executable scripts in this directory, (such as ./search,
+./reply, etc.)
-The easiest way to run tests is to say "make". This runs all
-the tests.
-
- *** t0000-basic.sh ***
- * ok 1: .git/objects should be empty after git-init in an empty repo.
- * ok 2: .git/objects should have 256 subdirectories.
- * ok 3: git-update-index without --add should fail adding.
- ...
- * ok 23: no diff after checkout and git-update-index --refresh.
- * passed all 23 test(s)
- *** t0100-environment-names.sh ***
- * ok 1: using old names should issue warnings.
- * ok 2: using old names but having new names should not issue warnings.
- ...
-
-Or you can run each test individually from command line, like
-this:
-
- $ sh ./t3001-ls-files-killed.sh
- * ok 1: git-update-index --add to add various paths.
- * ok 2: git-ls-files -k to show killed files.
- * ok 3: validate git-ls-files -k output.
- * passed all 3 test(s)
-
-You can pass --verbose (or -v), --debug (or -d), and --immediate
-(or -i) command line argument to the test, or by setting GIT_TEST_OPTS
-appropriately before running "make".
-
---verbose::
- This makes the test more verbose. Specifically, the
- command being run and their output if any are also
- output.
+The following command-line options are available when running tests:
--debug::
This may help the person who is developing a new test.
This causes the test to immediately exit upon the first
failed test.
---long-tests::
- This causes additional long-running tests to be run (where
- available), for more exhaustive testing.
-
--valgrind::
- Execute all Git binaries with valgrind and exit with status
+ Execute notmuch with valgrind and exit with status
126 on errors (just like regular tests, this will only stop
the test script when running under -i). Valgrind errors
go to stderr, so you might want to pass the -v option, too.
As the names depend on the tests' file names, it is safe to
run the tests with this option in parallel.
---with-dashes::
- By default tests are run without dashed forms of
- commands (like git-commit) in the PATH (it only uses
- wrappers from ../bin-wrappers). Use this option to include
- the build directory (..) in the PATH, which contains all
- the dashed forms of commands. This option is currently
- implied by other options like --valgrind and
- GIT_TEST_INSTALLED.
-
-You can also set the GIT_TEST_INSTALLED environment variable to
-the bindir of an existing git installation to test that installation.
-You still need to have built this git sandbox, from which various
-test-* support programs, templates, and perl libraries are used.
-If your installed git is incomplete, it will silently test parts of
-your built version instead.
-
-When using GIT_TEST_INSTALLED, you can also set GIT_TEST_EXEC_PATH to
-override the location of the dashed-form subcommands (what
-GIT_EXEC_PATH would be used for during normal operation).
-GIT_TEST_EXEC_PATH defaults to `$GIT_TEST_INSTALLED/git --exec-path`.
+When invoking the test suite via "make test" any of the above options
+can be specified as follows:
+ make test OPTIONS="--verbose"
Skipping Tests
--------------
+If, for any reason, you need to skip one or more tests, you can do so
+by setting the NOTMUCH_SKIP_TESTS variable to the name of one or more
+sections of tests.
-In some environments, certain tests have no way of succeeding
-due to platform limitation, such as lack of 'unzip' program, or
-filesystem that do not allow arbitrary sequence of non-NUL bytes
-as pathnames.
-
-You should be able to say something like
-
- $ GIT_SKIP_TESTS=t9200.8 sh ./t9200-git-cvsexport-commit.sh
-
-and even:
-
- $ GIT_SKIP_TESTS='t[0-4]??? t91?? t9200.8' make
-
-to omit such tests. The value of the environment variable is a
-SP separated list of patterns that tells which tests to skip,
-and either can match the "t[0-9]{4}" part to skip the whole
-test, or t[0-9]{4} followed by ".$number" to say which
-particular test to skip.
+For example:
-Note that some tests in the existing test suite rely on previous
-test item, so you cannot arbitrarily disable one and expect the
-remainder of test to check what the test originally was intended
-to check.
+ $ NOTMUCH_SKIP_TESTS="search reply" make test
+Even more fine-grained skipping is possible by appending a test number
+(or glob pattern) after the section name. For example, the first
+search test and the second reply test could be skipped with:
-Naming Tests
-------------
-
-The test files are named as:
-
- tNNNN-commandname-details.sh
-
-where N is a decimal digit.
-
-First digit tells the family:
-
- 0 - the absolute basics and global stuff
- 1 - the basic commands concerning database
- 2 - the basic commands concerning the working tree
- 3 - the other basic commands (e.g. ls-files)
- 4 - the diff commands
- 5 - the pull and exporting commands
- 6 - the revision tree commands (even e.g. merge-base)
- 7 - the porcelainish commands concerning the working tree
- 8 - the porcelainish commands concerning forensics
- 9 - the git tools
-
-Second digit tells the particular command we are testing.
-
-Third digit (optionally) tells the particular switch or group of switches
-we are testing.
-
-If you create files under t/ directory (i.e. here) that is not
-the top-level test script, never name the file to match the above
-pattern. The Makefile here considers all such files as the
-top-level test script and tries to run all of them. A care is
-especially needed if you are creating a common test library
-file, similar to test-lib.sh, because such a library file may
-not be suitable for standalone execution.
+ $ NOTMUCH_SKIP_TESTS="search.1 reply.2" make test
+Note that some tests in the existing test suite rely on previous test
+items, so you cannot arbitrarily skip any test and expect the
+remaining tests to be unaffected.
Writing Tests
-------------
-
-The test script is written as a shell script. It should start
-with the standard "#!/bin/bash" with copyright notices, and an
+The test script is written as a shell script. It should start with
+the standard "#!/usr/bin/env bash" with copyright notices, and an
assignment to variable 'test_description', like this:
- #!/bin/bash
+ #!/usr/bin/env bash
#
# Copyright (c) 2005 Junio C Hamano
#
test_description='xxx test (option --frotz)
- This test registers the following structure in the cache
- and tries to run git-ls-files with option --frotz.'
-
+ This test exercises the "notmuch xxx" command when
+ given the option --frotz.'
Source 'test-lib.sh'
--------------------
-
After assigning test_description, the test script should source
test-lib.sh like this:
- If the script is invoked with command line argument --help
(or -h), it shows the test_description and exits.
- - Creates a test directory with default notmuch-config and empty mail
- store. This directory is 't/trash directory.<test-basename>' (note
- the space) if you must know, but I do not think you care. The path
- to notmuch-config is exported in NOTMUCH_CONFIG environment
- variable and mail store path is stored in MAIL_DIR variable.
+ - Creates a temporary directory with default notmuch-config and a
+ mail store with a corpus of mail, (initially, 50 early messages
+ sent to the notmuch list). This directory is
+ test/tmp.<test-basename>. The path to notmuch-config is exported in
+ NOTMUCH_CONFIG environment variable and mail store path is stored
+ in MAIL_DIR variable.
- Defines standard test helper functions for your scripts to
use. These functions are designed to make all scripts behave
consistently when command line arguments --verbose (or -v),
--debug (or -d), and --immediate (or -i) is given.
-
End with test_done
------------------
-
Your script will be a sequence of tests, using helper functions
from the test harness library. At the end of the script, call
'test_done'.
-
Test harness library
--------------------
-
There are a handful helper functions defined in the test harness
library for your script to use.
- - test_expect_success <message> <script>
+ test_expect_success <message> <script>
This takes two strings as parameter, and evaluates the
<script>. If it yields success, test is considered
successful. <message> should state what it is testing.
- Example:
-
- test_expect_success \
- 'git-write-tree should be able to write an empty tree.' \
- 'tree=$(git-write-tree)'
-
- - test_expect_failure <message> <script>
+ test_expect_failure <message> <script>
This is NOT the opposite of test_expect_success, but is used
to mark a test that demonstrates a known breakage. Unlike
success and "still broken" on failure. Failures from these
tests won't cause -i (immediate) to stop.
- - test_debug <script>
+ test_begin_subtest <message>
+
+ Set the test description message for a subsequent test_expect_equal
+ invocation (see below).
+
+ test_expect_equal <output> <expected>
+
+ This is an often-used convenience function built on top of
+ test_expect_success. It uses the message from the last
+ test_begin_subtest call, so call before calling
+ test_expect_equal. This function generates a successful test if
+ both the <output> and <expected> strings are identical. If not, it
+ will generate a failure and print the difference of the two
+ strings.
+
+ test_expect_equal_failure <output> <expected>
+
+ This works similar to test_expect_equal (see above) but is used to
+ mark a test that demonstrates a known breakage, (that is, the
+ author of the test expects "output" and "expected" to differ until
+ the breakage is fixed). See test_expect_failure for details.
+
+ test_debug <script>
This takes a single argument, <script>, and evaluates it only
when the test script is started with --debug command line
argument. This is primarily meant for use during the
development of a new test script.
- - test_done
+ test_emacs <emacs-lisp-expressions>
+
+ This function executes the provided emacs lisp script within
+ emacs. The script can be a sequence of emacs lisp expressions,
+ (that is, they will be evaluated within a progn form). The lisp
+ expressions can call `message' to generate output on stdout to be
+ examined by the calling test script.
+
+ test_done
Your test script must have test_done at the end. Its purpose
is to summarize successes and failures in the test script and
exit with an appropriate error code.
- - test_tick
-
- Make commit and tag names consistent by setting the author and
- committer times to defined stated. Subsequent calls will
- advance the times by a fixed amount.
-
- - test_commit <message> [<filename> [<contents>]]
-
- Creates a commit with the given message, committing the given
- file with the given contents (default for both is to reuse the
- message string), and adds a tag (again reusing the message
- string as name). Calls test_tick to make the SHA-1s
- reproducible.
-
- - test_merge <message> <commit-or-tag>
-
- Merges the given rev using the given message. Like test_commit,
- creates a tag and calls test_tick before committing.
-
-Tips for Writing Tests
-----------------------
-
-As with any programming projects, existing programs are the best
-source of the information. However, do _not_ emulate
-t0000-basic.sh when writing your tests. The test is special in
-that it tries to validate the very core of GIT. For example, it
-knows that there will be 256 subdirectories under .git/objects/,
-and it knows that the object ID of an empty tree is a certain
-40-byte string. This is deliberately done so in t0000-basic.sh
-because the things the very basic core test tries to achieve is
-to serve as a basis for people who are changing the GIT internal
-drastically. For these people, after making certain changes,
-not seeing failures from the basic test _is_ a failure. And
-such drastic changes to the core GIT that even changes these
-otherwise supposedly stable object IDs should be accompanied by
-an update to t0000-basic.sh.
-
-However, other tests that simply rely on basic parts of the core
-GIT working properly should not have that level of intimate
-knowledge of the core GIT internals. If all the test scripts
-hardcoded the object IDs like t0000-basic.sh does, that defeats
-the purpose of t0000-basic.sh, which is to isolate that level of
-validation in one place. Your test also ends up needing
-updating when such a change to the internal happens, so do _not_
-do it and leave the low level of validation to t0000-basic.sh.
+There are also a number of mail-specific functions which are useful in
+writing tests:
+
+ generate_message
+
+ Generates a message with an optional template. Most tests will
+ actually prefer to call add_message. See below.
+
+ add_message
+
+ Generate a message and add it to the database (by calling "notmuch
+ new"). It is sufficient to simply call add_message with no
+ arguments if you don't care about the content of the message. If
+ more control is needed, arguments can be provide to specify many
+ different header values for the new message. See the documentation
+ within test-lib.sh or refer to many example calls within existing
+ tests.
+
+ add_email_corpus
+
+ This function should be called at the beginning of a test file
+ when a test needs to operate on a non-empty body of messages. It
+ will initialize the mail database to a known state of 50 sample
+ messages, (culled from the early history of the notmuch mailing
+ list).
projects / notmuch / blobdiff