X-Git-Url: https://git.notmuchmail.org/git?p=notmuch;a=blobdiff_plain;f=test%2FREADME;h=79a9b1b2f9a15aba7466e4e5b1b308663dd1296c;hp=af27d603d38dceb726f0e778652c9d23b612ffb6;hb=84719b08f757a6079f4c3331d0c476d19b265948;hpb=223987bacef4dcc51952084707d1a664765c7c6e

diff --git a/test/README b/test/README
index af27d603..79a9b1b2 100644
--- a/test/README
+++ b/test/README
@@ -1,51 +1,35 @@
-Core GIT Tests
-==============
+Notmuch test suite
+==================
+This directory contains the test suite for notmuch.
 
-This directory holds many test scripts for core GIT tools.  The
-first part of this short document describes how to run the tests
-and read their output.
+When fixing bugs or enhancing notmuch, you are strongly encouraged to
+add tests in this directory to cover what you are trying to fix or
+enhance.
 
-When fixing the tools or adding enhancements, you are strongly
-encouraged to add tests in this directory to cover what you are
-trying to fix or enhance.  The later part of this short document
-describes how your test scripts should be organized.
+Prerequisites
+-------------
+Some tests require external dependencies to run. Without them, they
+will be skipped, or (rarely) marked failed. Please install these, so
+that you know if you break anything.
 
+  - dtach(1)
+  - emacs(1)
+  - emacsclient(1)
+  - gdb(1)
+  - gpg(1)
+  - python(1)
 
 Running Tests
 -------------
+The easiest way to run tests is to say "make test", (or simply run the
+notmuch-test script). Either command will run all available tests.
+
+Alternately, you can run a specific subset of tests by simply invoking
+one of the executable scripts in this directory, (such as ./search,
+./reply, etc). Note that you will probably want "make test-binaries"
+before running individual tests.
 
-The easiest way to run tests is to say "make".  This runs all
-the tests.
-
-    *** t0000-basic.sh ***
-    *   ok 1: .git/objects should be empty after git-init in an empty repo.
-    *   ok 2: .git/objects should have 256 subdirectories.
-    *   ok 3: git-update-index without --add should fail adding.
-    ...
-    *   ok 23: no diff after checkout and git-update-index --refresh.
-    * passed all 23 test(s)
-    *** t0100-environment-names.sh ***
-    *   ok 1: using old names should issue warnings.
-    *   ok 2: using old names but having new names should not issue warnings.
-    ...
-
-Or you can run each test individually from command line, like
-this:
-
-    $ sh ./t3001-ls-files-killed.sh
-    *   ok 1: git-update-index --add to add various paths.
-    *   ok 2: git-ls-files -k to show killed files.
-    *   ok 3: validate git-ls-files -k output.
-    * passed all 3 test(s)
-
-You can pass --verbose (or -v), --debug (or -d), and --immediate
-(or -i) command line argument to the test, or by setting GIT_TEST_OPTS
-appropriately before running "make".
-
---verbose::
-	This makes the test more verbose.  Specifically, the
-	command being run and their output if any are also
-	output.
+The following command-line options are available when running tests:
 
 --debug::
 	This may help the person who is developing a new test.
@@ -55,12 +39,8 @@ appropriately before running "make".
 	This causes the test to immediately exit upon the first
 	failed test.
 
---long-tests::
-	This causes additional long-running tests to be run (where
-	available), for more exhaustive testing.
-
 --valgrind::
-	Execute all Git binaries with valgrind and exit with status
+	Execute notmuch with valgrind and exit with status
 	126 on errors (just like regular tests, this will only stop
 	the test script when running under -i).  Valgrind errors
 	go to stderr, so you might want to pass the -v option, too.
@@ -75,113 +55,73 @@ appropriately before running "make".
 	As the names depend on the tests' file names, it is safe to
 	run the tests with this option in parallel.
 
---with-dashes::
-	By default tests are run without dashed forms of
-	commands (like git-commit) in the PATH (it only uses
-	wrappers from ../bin-wrappers).  Use this option to include
-	the build directory (..) in the PATH, which contains all
-	the dashed forms of commands.  This option is currently
-	implied by other options like --valgrind and
-	GIT_TEST_INSTALLED.
-
-You can also set the GIT_TEST_INSTALLED environment variable to
-the bindir of an existing git installation to test that installation.
-You still need to have built this git sandbox, from which various
-test-* support programs, templates, and perl libraries are used.
-If your installed git is incomplete, it will silently test parts of
-your built version instead.
-
-When using GIT_TEST_INSTALLED, you can also set GIT_TEST_EXEC_PATH to
-override the location of the dashed-form subcommands (what
-GIT_EXEC_PATH would be used for during normal operation).
-GIT_TEST_EXEC_PATH defaults to `$GIT_TEST_INSTALLED/git --exec-path`.
-
-
-Skipping Tests
---------------
-
-In some environments, certain tests have no way of succeeding
-due to platform limitation, such as lack of 'unzip' program, or
-filesystem that do not allow arbitrary sequence of non-NUL bytes
-as pathnames.
-
-You should be able to say something like
+--root=<dir>::
+	This runs the testsuites specified under a separate directory.
+	However, caution is advised, as not all tests are maintained
+	with this relocation in mind, so some tests may behave
+	differently.
 
-    $ GIT_SKIP_TESTS=t9200.8 sh ./t9200-git-cvsexport-commit.sh
+	Pointing this argument at a tmpfs filesystem can improve the
+	speed of the test suite for some users.
 
-and even:
+When invoking the test suite via "make test" any of the above options
+can be specified as follows:
 
-    $ GIT_SKIP_TESTS='t[0-4]??? t91?? t9200.8' make
+	make test OPTIONS="--verbose"
 
-to omit such tests.  The value of the environment variable is a
-SP separated list of patterns that tells which tests to skip,
-and either can match the "t[0-9]{4}" part to skip the whole
-test, or t[0-9]{4} followed by ".$number" to say which
-particular test to skip.
+You can choose an emacs binary (and corresponding emacsclient) to run
+the tests in one of the following ways.
 
-Note that some tests in the existing test suite rely on previous
-test item, so you cannot arbitrarily disable one and expect the
-remainder of test to check what the test originally was intended
-to check.
+	TEST_EMACS=my-special-emacs TEST_EMACSCLIENT=my-emacsclient make test
+	TEST_EMACS=my-special-emacs TEST_EMACSCLIENT=my-emacsclient ./emacs
+	make test TEST_EMACS=my-special-emacs TEST_EMACSCLIENT=my-emacsclient
 
+Quiet Execution
+---------------
 
-Naming Tests
-------------
+Normally, when new script starts and when test PASSes you get a message
+printed on screen. This printing can be disabled by setting the
+NOTMUCH_TEST_QUIET variable to a non-null value. Message on test
+failures and skips are still printed.
 
-The test files are named as:
-
-	tNNNN-commandname-details.sh
-
-where N is a decimal digit.
-
-First digit tells the family:
+Skipping Tests
+--------------
+If, for any reason, you need to skip one or more tests, you can do so
+by setting the NOTMUCH_SKIP_TESTS variable to the name of one or more
+sections of tests.
 
-	0 - the absolute basics and global stuff
-	1 - the basic commands concerning database
-	2 - the basic commands concerning the working tree
-	3 - the other basic commands (e.g. ls-files)
-	4 - the diff commands
-	5 - the pull and exporting commands
-	6 - the revision tree commands (even e.g. merge-base)
-	7 - the porcelainish commands concerning the working tree
-	8 - the porcelainish commands concerning forensics
-	9 - the git tools
+For example:
 
-Second digit tells the particular command we are testing.
+    $ NOTMUCH_SKIP_TESTS="search reply" make test
 
-Third digit (optionally) tells the particular switch or group of switches
-we are testing.
+Even more fine-grained skipping is possible by appending a test number
+(or glob pattern) after the section name. For example, the first
+search test and the second reply test could be skipped with:
 
-If you create files under t/ directory (i.e. here) that is not
-the top-level test script, never name the file to match the above
-pattern.  The Makefile here considers all such files as the
-top-level test script and tries to run all of them.  A care is
-especially needed if you are creating a common test library
-file, similar to test-lib.sh, because such a library file may
-not be suitable for standalone execution.
+    $ NOTMUCH_SKIP_TESTS="search.1 reply.2" make test
 
+Note that some tests in the existing test suite rely on previous test
+items, so you cannot arbitrarily skip any test and expect the
+remaining tests to be unaffected.
 
 Writing Tests
 -------------
-
-The test script is written as a shell script.  It should start
-with the standard "#!/bin/bash" with copyright notices, and an
+The test script is written as a shell script.  It should start with
+the standard "#!/usr/bin/env bash" with copyright notices, and an
 assignment to variable 'test_description', like this:
 
-	#!/bin/bash
+	#!/usr/bin/env bash
 	#
 	# Copyright (c) 2005 Junio C Hamano
 	#
 
 	test_description='xxx test (option --frotz)
 
-	This test registers the following structure in the cache
-	and tries to run git-ls-files with option --frotz.'
-
+	This test exercises the "notmuch xxx" command when
+	given the option --frotz.'
 
 Source 'test-lib.sh'
 --------------------
-
 After assigning test_description, the test script should source
 test-lib.sh like this:
 
@@ -192,108 +132,152 @@ This test harness library does the following things:
  - If the script is invoked with command line argument --help
    (or -h), it shows the test_description and exits.
 
- - Creates a test directory with default notmuch-config and empty mail
-   store. This directory is 't/trash directory.<test-basename>' (note
-   the space) if you must know, but I do not think you care. The path
-   to notmuch-config is exported in NOTMUCH_CONFIG environment
-   variable and mail store path is stored in MAIL_DIR variable.
+ - Creates a temporary directory with default notmuch-config and a
+   mail store with a corpus of mail, (initially, 50 early messages
+   sent to the notmuch list). This directory is
+   test/tmp.<test-basename>. The path to notmuch-config is exported in
+   NOTMUCH_CONFIG environment variable and mail store path is stored
+   in MAIL_DIR variable.
 
  - Defines standard test helper functions for your scripts to
    use.  These functions are designed to make all scripts behave
    consistently when command line arguments --verbose (or -v),
    --debug (or -d), and --immediate (or -i) is given.
 
-
 End with test_done
 ------------------
-
 Your script will be a sequence of tests, using helper functions
 from the test harness library.  At the end of the script, call
 'test_done'.
 
-
 Test harness library
 --------------------
-
 There are a handful helper functions defined in the test harness
 library for your script to use.
 
- - test_expect_success <message> <script>
+ test_expect_success <message> <script>
 
    This takes two strings as parameter, and evaluates the
    <script>.  If it yields success, test is considered
    successful.  <message> should state what it is testing.
 
-   Example:
+ test_begin_subtest <message>
+
+   Set the test description message for a subsequent test_expect_equal
+   invocation (see below).
+
+ test_subtest_known_broken
+
+   Mark the current test as broken.  Such tests are expected to fail.
+   Unlike the normal tests, which say "PASS" on success and "FAIL" on
+   failure, these will say "FIXED" on success and "BROKEN" on failure.
+   Failures from these tests won't cause -i (immediate) to stop.  A
+   test must call this before any test_expect_* function.
+
+ test_expect_equal <output> <expected>
+
+   This is an often-used convenience function built on top of
+   test_expect_success. It uses the message from the last
+   test_begin_subtest call, so call before calling
+   test_expect_equal. This function generates a successful test if
+   both the <output> and <expected> strings are identical. If not, it
+   will generate a failure and print the difference of the two
+   strings.
 
-	test_expect_success \
-	    'git-write-tree should be able to write an empty tree.' \
-	    'tree=$(git-write-tree)'
+ test_expect_equal_file <file1> <file2>
 
- - test_expect_failure <message> <script>
+   Identical to test_expect_equal, except that <file1> and <file2>
+   are files instead of strings.  This is a much more robust method to
+   compare formatted textual information, since it also notices
+   whitespace and closing newline differences.
 
-   This is NOT the opposite of test_expect_success, but is used
-   to mark a test that demonstrates a known breakage.  Unlike
-   the usual test_expect_success tests, which say "ok" on
-   success and "FAIL" on failure, this will say "FIXED" on
-   success and "still broken" on failure.  Failures from these
-   tests won't cause -i (immediate) to stop.
+ test_expect_equal_json <output> <expected>
 
- - test_debug <script>
+   Identical to test_expect_equal, except that the two strings are
+   treated as JSON and canonicalized before equality testing.  This is
+   useful to abstract away from whitespace differences in the expected
+   output and that generated by running a notmuch command.
+
+ test_debug <script>
 
    This takes a single argument, <script>, and evaluates it only
    when the test script is started with --debug command line
    argument.  This is primarily meant for use during the
    development of a new test script.
 
- - test_done
+ test_emacs <emacs-lisp-expressions>
+
+   This function executes the provided emacs lisp script within
+   emacs. The script can be a sequence of emacs lisp expressions,
+   (that is, they will be evaluated within a progn form). Emacs
+   stdout and stderr is not available, the common way to get output
+   is to save it to a file. There are some auxiliary functions
+   useful in emacs tests provided in test-lib.el. Do not use `setq'
+   for setting variables in Emacs tests because it affects other
+   tests that may run in the same Emacs instance.  Use `let' instead
+   so the scope of the changed variables is limited to a single test.
+
+ test_emacs_expect_t <emacs-lisp-expressions>
+
+  This function executes the provided emacs lisp script within
+  emacs in a manner similar to 'test_emacs'. The expressions should
+  return the value `t' to indicate that the test has passed. If the
+  test does not return `t' then it is considered failed and all data
+  returned by the test is reported to the tester.
+
+ test_done
 
    Your test script must have test_done at the end.  Its purpose
    is to summarize successes and failures in the test script and
    exit with an appropriate error code.
 
- - test_tick
-
-   Make commit and tag names consistent by setting the author and
-   committer times to defined stated.  Subsequent calls will
-   advance the times by a fixed amount.
-
- - test_commit <message> [<filename> [<contents>]]
-
-   Creates a commit with the given message, committing the given
-   file with the given contents (default for both is to reuse the
-   message string), and adds a tag (again reusing the message
-   string as name).  Calls test_tick to make the SHA-1s
-   reproducible.
-
- - test_merge <message> <commit-or-tag>
-
-   Merges the given rev using the given message.  Like test_commit,
-   creates a tag and calls test_tick before committing.
-
-Tips for Writing Tests
-----------------------
-
-As with any programming projects, existing programs are the best
-source of the information.  However, do _not_ emulate
-t0000-basic.sh when writing your tests.  The test is special in
-that it tries to validate the very core of GIT.  For example, it
-knows that there will be 256 subdirectories under .git/objects/,
-and it knows that the object ID of an empty tree is a certain
-40-byte string.  This is deliberately done so in t0000-basic.sh
-because the things the very basic core test tries to achieve is
-to serve as a basis for people who are changing the GIT internal
-drastically.  For these people, after making certain changes,
-not seeing failures from the basic test _is_ a failure.  And
-such drastic changes to the core GIT that even changes these
-otherwise supposedly stable object IDs should be accompanied by
-an update to t0000-basic.sh.
-
-However, other tests that simply rely on basic parts of the core
-GIT working properly should not have that level of intimate
-knowledge of the core GIT internals.  If all the test scripts
-hardcoded the object IDs like t0000-basic.sh does, that defeats
-the purpose of t0000-basic.sh, which is to isolate that level of
-validation in one place.  Your test also ends up needing
-updating when such a change to the internal happens, so do _not_
-do it and leave the low level of validation to t0000-basic.sh.
+There are also a number of notmuch-specific auxiliary functions and
+variables which are useful in writing tests:
+
+  generate_message
+
+    Generates a message with an optional template. Most tests will
+    actually prefer to call add_message. See below.
+
+  add_message
+
+    Generate a message and add it to the database (by calling "notmuch
+    new"). It is sufficient to simply call add_message with no
+    arguments if you don't care about the content of the message. If
+    more control is needed, arguments can be provide to specify many
+    different header values for the new message. See the documentation
+    within test-lib.sh or refer to many example calls within existing
+    tests.
+
+  add_email_corpus
+
+    This function should be called at the beginning of a test file
+    when a test needs to operate on a non-empty body of messages. It
+    will initialize the mail database to a known state of 50 sample
+    messages, (culled from the early history of the notmuch mailing
+    list).
+
+  notmuch_counter_reset
+  $notmuch_counter_command
+  notmuch_counter_value
+
+    These allow to count how many times notmuch binary is called.
+    notmuch_counter_reset() function generates a script that counts
+    how many times it is called and resets the counter to zero.  The
+    function sets $notmuch_counter_command variable to the path to the
+    generated script that should be called instead of notmuch to do
+    the counting.  The notmuch_counter_value() function prints the
+    current counter value.
+
+There are also functions which remove various environment-dependent
+values from notmuch output; these are useful to ensure that test
+results remain consistent across different machines.
+
+ notmuch_search_sanitize
+ notmuch_show_sanitize
+ notmuch_show_sanitize_all
+ notmuch_json_show_sanitize
+
+   All these functions should receive the text to be sanitized as the
+   input of a pipe, e.g.
+   output=`notmuch search "..." | notmuch_search_sanitize`