X-Git-Url: https://git.notmuchmail.org/git?p=notmuch;a=blobdiff_plain;f=test%2FREADME;h=104a120ea28bfb7250243f1d331ed8efd1e99fa0;hp=dcd3ebb5f2dcdbf15ca0e4a043b45cd2fc36cbb5;hb=b183f2635eaf323066f26c7ca22d931d4f4ccd3a;hpb=0083854b1204f077e98b1d9c4ecfa2a4844ee716 diff --git a/test/README b/test/README index dcd3ebb5..104a120e 100644 --- a/test/README +++ b/test/README @@ -1,51 +1,49 @@ -Core GIT Tests -============== +Notmuch test suite +================== +This directory contains the test suite for notmuch. -This directory holds many test scripts for core GIT tools. The -first part of this short document describes how to run the tests -and read their output. +When fixing bugs or enhancing notmuch, you are strongly encouraged to +add tests in this directory to cover what you are trying to fix or +enhance. -When fixing the tools or adding enhancements, you are strongly -encouraged to add tests in this directory to cover what you are -trying to fix or enhance. The later part of this short document -describes how your test scripts should be organized. +Prerequisites +------------- +The test system itself requires: + + - bash(1) version 4.0 or newer + +Without bash 4.0+ the tests just refuse to run. + +Some tests require external dependencies to run. Without them, they +will be skipped, or (rarely) marked failed. Please install these, so +that you know if you break anything. + + - GNU tar(1) + - dtach(1) + - emacs(1) + - emacsclient(1) + - gdb(1) + - gpg(1) + - python(1) + +If your system lacks these tools or have older, non-upgreable versions +of these, please (possibly compile and) install these to some other +path, for example /usr/local/bin or /opt/gnu/bin. Then prepend the +chosen directory to your PATH before running the tests. +e.g. env PATH=/opt/gnu/bin:$PATH make test Running Tests ------------- +The easiest way to run tests is to say "make test", (or simply run the +notmuch-test script). Either command will run all available tests. -The easiest way to run tests is to say "make". This runs all -the tests. - - *** t0000-basic.sh *** - * ok 1: .git/objects should be empty after git-init in an empty repo. - * ok 2: .git/objects should have 256 subdirectories. - * ok 3: git-update-index without --add should fail adding. - ... - * ok 23: no diff after checkout and git-update-index --refresh. - * passed all 23 test(s) - *** t0100-environment-names.sh *** - * ok 1: using old names should issue warnings. - * ok 2: using old names but having new names should not issue warnings. - ... - -Or you can run each test individually from command line, like -this: - - $ sh ./t3001-ls-files-killed.sh - * ok 1: git-update-index --add to add various paths. - * ok 2: git-ls-files -k to show killed files. - * ok 3: validate git-ls-files -k output. - * passed all 3 test(s) - -You can pass --verbose (or -v), --debug (or -d), and --immediate -(or -i) command line argument to the test, or by setting GIT_TEST_OPTS -appropriately before running "make". - ---verbose:: - This makes the test more verbose. Specifically, the - command being run and their output if any are also - output. +Alternately, you can run a specific subset of tests by simply invoking +one of the executable scripts in this directory, (such as ./T*-search.sh, +./T*-reply.sh, etc). Note that you will probably want "make test-binaries" +before running individual tests. + +The following command-line options are available when running tests: --debug:: This may help the person who is developing a new test. @@ -55,12 +53,8 @@ appropriately before running "make". This causes the test to immediately exit upon the first failed test. ---long-tests:: - This causes additional long-running tests to be run (where - available), for more exhaustive testing. - --valgrind:: - Execute all Git binaries with valgrind and exit with status + Execute notmuch with valgrind and exit with status 126 on errors (just like regular tests, this will only stop the test script when running under -i). Valgrind errors go to stderr, so you might want to pass the -v option, too. @@ -75,223 +69,249 @@ appropriately before running "make". As the names depend on the tests' file names, it is safe to run the tests with this option in parallel. ---with-dashes:: - By default tests are run without dashed forms of - commands (like git-commit) in the PATH (it only uses - wrappers from ../bin-wrappers). Use this option to include - the build directory (..) in the PATH, which contains all - the dashed forms of commands. This option is currently - implied by other options like --valgrind and - GIT_TEST_INSTALLED. - -You can also set the GIT_TEST_INSTALLED environment variable to -the bindir of an existing git installation to test that installation. -You still need to have built this git sandbox, from which various -test-* support programs, templates, and perl libraries are used. -If your installed git is incomplete, it will silently test parts of -your built version instead. +--root=:: + This runs the testsuites specified under a separate directory. + However, caution is advised, as not all tests are maintained + with this relocation in mind, so some tests may behave + differently. -When using GIT_TEST_INSTALLED, you can also set GIT_TEST_EXEC_PATH to -override the location of the dashed-form subcommands (what -GIT_EXEC_PATH would be used for during normal operation). -GIT_TEST_EXEC_PATH defaults to `$GIT_TEST_INSTALLED/git --exec-path`. + Pointing this argument at a tmpfs filesystem can improve the + speed of the test suite for some users. +Certain tests require precomputed databases to complete. You can fetch these +databases with -Skipping Tests --------------- + make download-test-databases -In some environments, certain tests have no way of succeeding -due to platform limitation, such as lack of 'unzip' program, or -filesystem that do not allow arbitrary sequence of non-NUL bytes -as pathnames. +If you do not download the test databases, the relevant tests will be +skipped. -You should be able to say something like +When invoking the test suite via "make test" any of the above options +can be specified as follows: - $ GIT_SKIP_TESTS=t9200.8 sh ./t9200-git-cvsexport-commit.sh + make test OPTIONS="--verbose" -and even: +You can choose an emacs binary (and corresponding emacsclient) to run +the tests in one of the following ways. - $ GIT_SKIP_TESTS='t[0-4]??? t91?? t9200.8' make + TEST_EMACS=my-emacs TEST_EMACSCLIENT=my-emacsclient make test + TEST_EMACS=my-emacs TEST_EMACSCLIENT=my-emacsclient ./T*-emacs.sh + make test TEST_EMACS=my-emacs TEST_EMACSCLIENT=my-emacsclient -to omit such tests. The value of the environment variable is a -SP separated list of patterns that tells which tests to skip, -and either can match the "t[0-9]{4}" part to skip the whole -test, or t[0-9]{4} followed by ".$number" to say which -particular test to skip. +Some tests may require a c compiler. You can choose the name and flags similarly +to with emacs, e.g. -Note that some tests in the existing test suite rely on previous -test item, so you cannot arbitrarily disable one and expect the -remainder of test to check what the test originally was intended -to check. + make test TEST_CC=gcc TEST_CFLAGS="-g -O2" +Quiet Execution +--------------- -Naming Tests ------------- +Normally, when new script starts and when test PASSes you get a message +printed on screen. This printing can be disabled by setting the +NOTMUCH_TEST_QUIET variable to a non-null value. Message on test +failures and skips are still printed. -The test files are named as: - - tNNNN-commandname-details.sh - -where N is a decimal digit. +Skipping Tests +-------------- +If, for any reason, you need to skip one or more tests, you can do so +by setting the NOTMUCH_SKIP_TESTS variable to the name of one or more +sections of tests. -First digit tells the family: +For example: - 0 - the absolute basics and global stuff - 1 - the basic commands concerning database - 2 - the basic commands concerning the working tree - 3 - the other basic commands (e.g. ls-files) - 4 - the diff commands - 5 - the pull and exporting commands - 6 - the revision tree commands (even e.g. merge-base) - 7 - the porcelainish commands concerning the working tree - 8 - the porcelainish commands concerning forensics - 9 - the git tools + $ NOTMUCH_SKIP_TESTS="search reply" make test -Second digit tells the particular command we are testing. +Even more fine-grained skipping is possible by appending a test number +(or glob pattern) after the section name. For example, the first +search test and the second reply test could be skipped with: -Third digit (optionally) tells the particular switch or group of switches -we are testing. + $ NOTMUCH_SKIP_TESTS="search.1 reply.2" make test -If you create files under t/ directory (i.e. here) that is not -the top-level test script, never name the file to match the above -pattern. The Makefile here considers all such files as the -top-level test script and tries to run all of them. A care is -especially needed if you are creating a common test library -file, similar to test-lib.sh, because such a library file may -not be suitable for standalone execution. +Note that some tests in the existing test suite rely on previous test +items, so you cannot arbitrarily skip any test and expect the +remaining tests to be unaffected. +Currently we do not consider skipped tests as build failures. For +maximum robustness, when setting up automated build processes, you +should explicitely skip tests, rather than relying on notmuch's +detection of missing prerequisites. In the future we may treat tests +unable to run because of missing prerequisites, but not explicitely +skipped by the user, as failures. Writing Tests ------------- +The test script is written as a shell script. It is to be named as +Tddd-testname.sh where 'ddd' is three digits and 'testname' the "bare" +name of your test. Tests will be run in order the 'ddd' part determines. -The test script is written as a shell script. It should start -with the standard "#!/bin/sh" with copyright notices, and an -assignment to variable 'test_description', like this: +The test script should start with the standard "#!/usr/bin/env bash" +and an assignment to variable 'test_description', like this: - #!/bin/sh - # - # Copyright (c) 2005 Junio C Hamano - # + #!/usr/bin/env bash test_description='xxx test (option --frotz) - This test registers the following structure in the cache - and tries to run git-ls-files with option --frotz.' - + This test exercises the "notmuch xxx" command when + given the option --frotz.' Source 'test-lib.sh' -------------------- - After assigning test_description, the test script should source test-lib.sh like this: - . ./test-lib.sh + . ./test-lib.sh || exit 1 This test harness library does the following things: - If the script is invoked with command line argument --help (or -h), it shows the test_description and exits. - - Creates an empty test directory with an empty .git/objects - database and chdir(2) into it. This directory is 't/trash directory' - if you must know, but I do not think you care. + - Creates a temporary directory with default notmuch-config and a + mail store with a corpus of mail, (initially, 50 early messages + sent to the notmuch list). This directory is + test/tmp.. The path to notmuch-config is exported in + NOTMUCH_CONFIG environment variable and mail store path is stored + in MAIL_DIR variable. - Defines standard test helper functions for your scripts to use. These functions are designed to make all scripts behave consistently when command line arguments --verbose (or -v), --debug (or -d), and --immediate (or -i) is given. - End with test_done ------------------ - Your script will be a sequence of tests, using helper functions from the test harness library. At the end of the script, call 'test_done'. - Test harness library -------------------- - There are a handful helper functions defined in the test harness library for your script to use. - - test_expect_success