X-Git-Url: https://git.notmuchmail.org/git?a=blobdiff_plain;f=test%2FREADME;h=43656a35baf9f9bc5e5d41aca2b65c8e4e763450;hb=512df7ec47a378f9807dd2c19db084fe6e4010ad;hp=af27d603d38dceb726f0e778652c9d23b612ffb6;hpb=223987bacef4dcc51952084707d1a664765c7c6e;p=notmuch
diff --git a/test/README b/test/README
index af27d603..43656a35 100644
--- a/test/README
+++ b/test/README
@@ -1,51 +1,35 @@
-Core GIT Tests
-==============
+Notmuch test suite
+==================
+This directory contains the test suite for notmuch.
-This directory holds many test scripts for core GIT tools. The
-first part of this short document describes how to run the tests
-and read their output.
+When fixing bugs or enhancing notmuch, you are strongly encouraged to
+add tests in this directory to cover what you are trying to fix or
+enhance.
-When fixing the tools or adding enhancements, you are strongly
-encouraged to add tests in this directory to cover what you are
-trying to fix or enhance. The later part of this short document
-describes how your test scripts should be organized.
+Prerequisites
+-------------
+Some tests require external dependencies to run. Without them, they
+will be skipped, or (rarely) marked failed. Please install these, so
+that you know if you break anything.
+ - dtach(1)
+ - emacs(1)
+ - emacsclient(1)
+ - gdb(1)
+ - gpg(1)
+ - python(1)
Running Tests
-------------
+The easiest way to run tests is to say "make test", (or simply run the
+notmuch-test script). Either command will run all available tests.
+
+Alternately, you can run a specific subset of tests by simply invoking
+one of the executable scripts in this directory, (such as ./search,
+./reply, etc). Note that you will probably want "make test-binaries"
+before running individual tests.
-The easiest way to run tests is to say "make". This runs all
-the tests.
-
- *** t0000-basic.sh ***
- * ok 1: .git/objects should be empty after git-init in an empty repo.
- * ok 2: .git/objects should have 256 subdirectories.
- * ok 3: git-update-index without --add should fail adding.
- ...
- * ok 23: no diff after checkout and git-update-index --refresh.
- * passed all 23 test(s)
- *** t0100-environment-names.sh ***
- * ok 1: using old names should issue warnings.
- * ok 2: using old names but having new names should not issue warnings.
- ...
-
-Or you can run each test individually from command line, like
-this:
-
- $ sh ./t3001-ls-files-killed.sh
- * ok 1: git-update-index --add to add various paths.
- * ok 2: git-ls-files -k to show killed files.
- * ok 3: validate git-ls-files -k output.
- * passed all 3 test(s)
-
-You can pass --verbose (or -v), --debug (or -d), and --immediate
-(or -i) command line argument to the test, or by setting GIT_TEST_OPTS
-appropriately before running "make".
-
---verbose::
- This makes the test more verbose. Specifically, the
- command being run and their output if any are also
- output.
+The following command-line options are available when running tests:
--debug::
This may help the person who is developing a new test.
@@ -55,12 +39,8 @@ appropriately before running "make".
This causes the test to immediately exit upon the first
failed test.
---long-tests::
- This causes additional long-running tests to be run (where
- available), for more exhaustive testing.
-
--valgrind::
- Execute all Git binaries with valgrind and exit with status
+ Execute notmuch with valgrind and exit with status
126 on errors (just like regular tests, this will only stop
the test script when running under -i). Valgrind errors
go to stderr, so you might want to pass the -v option, too.
@@ -75,113 +55,65 @@ appropriately before running "make".
As the names depend on the tests' file names, it is safe to
run the tests with this option in parallel.
---with-dashes::
- By default tests are run without dashed forms of
- commands (like git-commit) in the PATH (it only uses
- wrappers from ../bin-wrappers). Use this option to include
- the build directory (..) in the PATH, which contains all
- the dashed forms of commands. This option is currently
- implied by other options like --valgrind and
- GIT_TEST_INSTALLED.
-
-You can also set the GIT_TEST_INSTALLED environment variable to
-the bindir of an existing git installation to test that installation.
-You still need to have built this git sandbox, from which various
-test-* support programs, templates, and perl libraries are used.
-If your installed git is incomplete, it will silently test parts of
-your built version instead.
-
-When using GIT_TEST_INSTALLED, you can also set GIT_TEST_EXEC_PATH to
-override the location of the dashed-form subcommands (what
-GIT_EXEC_PATH would be used for during normal operation).
-GIT_TEST_EXEC_PATH defaults to `$GIT_TEST_INSTALLED/git --exec-path`.
-
-
-Skipping Tests
---------------
-
-In some environments, certain tests have no way of succeeding
-due to platform limitation, such as lack of 'unzip' program, or
-filesystem that do not allow arbitrary sequence of non-NUL bytes
-as pathnames.
-
-You should be able to say something like
+--root=
::
+ This runs the testsuites specified under a seperate directory.
+ However, caution is advised, as not all tests are maintained
+ with this relocation in mind, so some tests may behave
+ differently.
- $ GIT_SKIP_TESTS=t9200.8 sh ./t9200-git-cvsexport-commit.sh
+ Pointing this argument at a tmpfs filesystem can improve the
+ speed of the test suite for some users.
-and even:
+When invoking the test suite via "make test" any of the above options
+can be specified as follows:
- $ GIT_SKIP_TESTS='t[0-4]??? t91?? t9200.8' make
+ make test OPTIONS="--verbose"
-to omit such tests. The value of the environment variable is a
-SP separated list of patterns that tells which tests to skip,
-and either can match the "t[0-9]{4}" part to skip the whole
-test, or t[0-9]{4} followed by ".$number" to say which
-particular test to skip.
+You can choose an emacs binary to run the tests in one of the
+following ways.
-Note that some tests in the existing test suite rely on previous
-test item, so you cannot arbitrarily disable one and expect the
-remainder of test to check what the test originally was intended
-to check.
+ TEST_EMACS=my-special-emacs make test
+ TEST_EMACS=my-special-emacs ./emacs
+ make test TEST_EMACS=my-special-emacs
+Skipping Tests
+--------------
+If, for any reason, you need to skip one or more tests, you can do so
+by setting the NOTMUCH_SKIP_TESTS variable to the name of one or more
+sections of tests.
-Naming Tests
-------------
-
-The test files are named as:
-
- tNNNN-commandname-details.sh
-
-where N is a decimal digit.
-
-First digit tells the family:
-
- 0 - the absolute basics and global stuff
- 1 - the basic commands concerning database
- 2 - the basic commands concerning the working tree
- 3 - the other basic commands (e.g. ls-files)
- 4 - the diff commands
- 5 - the pull and exporting commands
- 6 - the revision tree commands (even e.g. merge-base)
- 7 - the porcelainish commands concerning the working tree
- 8 - the porcelainish commands concerning forensics
- 9 - the git tools
+For example:
-Second digit tells the particular command we are testing.
+ $ NOTMUCH_SKIP_TESTS="search reply" make test
-Third digit (optionally) tells the particular switch or group of switches
-we are testing.
+Even more fine-grained skipping is possible by appending a test number
+(or glob pattern) after the section name. For example, the first
+search test and the second reply test could be skipped with:
-If you create files under t/ directory (i.e. here) that is not
-the top-level test script, never name the file to match the above
-pattern. The Makefile here considers all such files as the
-top-level test script and tries to run all of them. A care is
-especially needed if you are creating a common test library
-file, similar to test-lib.sh, because such a library file may
-not be suitable for standalone execution.
+ $ NOTMUCH_SKIP_TESTS="search.1 reply.2" make test
+Note that some tests in the existing test suite rely on previous test
+items, so you cannot arbitrarily skip any test and expect the
+remaining tests to be unaffected.
Writing Tests
-------------
-
-The test script is written as a shell script. It should start
-with the standard "#!/bin/bash" with copyright notices, and an
+The test script is written as a shell script. It should start with
+the standard "#!/usr/bin/env bash" with copyright notices, and an
assignment to variable 'test_description', like this:
- #!/bin/bash
+ #!/usr/bin/env bash
#
# Copyright (c) 2005 Junio C Hamano
#
test_description='xxx test (option --frotz)
- This test registers the following structure in the cache
- and tries to run git-ls-files with option --frotz.'
-
+ This test exercises the "notmuch xxx" command when
+ given the option --frotz.'
Source 'test-lib.sh'
--------------------
-
After assigning test_description, the test script should source
test-lib.sh like this:
@@ -192,108 +124,132 @@ This test harness library does the following things:
- If the script is invoked with command line argument --help
(or -h), it shows the test_description and exits.
- - Creates a test directory with default notmuch-config and empty mail
- store. This directory is 't/trash directory.' (note
- the space) if you must know, but I do not think you care. The path
- to notmuch-config is exported in NOTMUCH_CONFIG environment
- variable and mail store path is stored in MAIL_DIR variable.
+ - Creates a temporary directory with default notmuch-config and a
+ mail store with a corpus of mail, (initially, 50 early messages
+ sent to the notmuch list). This directory is
+ test/tmp.. The path to notmuch-config is exported in
+ NOTMUCH_CONFIG environment variable and mail store path is stored
+ in MAIL_DIR variable.
- Defines standard test helper functions for your scripts to
use. These functions are designed to make all scripts behave
consistently when command line arguments --verbose (or -v),
--debug (or -d), and --immediate (or -i) is given.
-
End with test_done
------------------
-
Your script will be a sequence of tests, using helper functions
from the test harness library. At the end of the script, call
'test_done'.
-
Test harness library
--------------------
-
There are a handful helper functions defined in the test harness
library for your script to use.
- - test_expect_success