X-Git-Url: https://git.notmuchmail.org/git?p=notmuch;a=blobdiff_plain;f=test%2FREADME;h=8e06f44241a667609fdd5b47eda05b51e6fd3b47;hp=81c232ddd0a3fc90e1dbaa74f34f1252da004193;hb=87bdfbc91f65cb1031ef0ac8a804759f2061ac10;hpb=d59d9c81522d9127dde089ae9457f44de53f28e9 diff --git a/test/README b/test/README index 81c232dd..8e06f442 100644 --- a/test/README +++ b/test/README @@ -8,10 +8,17 @@ enhance. Prerequisites ------------- +The test system itself requires: + + - bash(1) version 4.0 or newer + +Without bash 4.0+ the tests just refuse to run. + Some tests require external dependencies to run. Without them, they will be skipped, or (rarely) marked failed. Please install these, so that you know if you break anything. + - GNU tar(1) - dtach(1) - emacs(1) - emacsclient(1) @@ -19,14 +26,32 @@ that you know if you break anything. - gpg(1) - python(1) +If your system lacks these tools or have older, non-upgradable versions +of these, please (possibly compile and) install these to some other +path, for example /usr/local/bin or /opt/gnu/bin. Then prepend the +chosen directory to your PATH before running the tests. + +e.g. env PATH=/opt/gnu/bin:$PATH make test + +For FreeBSD you need to install latest gdb from ports or packages and +provide path to it in TEST_GDB environment variable before executing +the tests, native FreeBSD gdb does not not work. If you install +coreutils, which provides GNU versions of basic utils like 'date' and +'base64' on FreeBSD, the test suite will use these instead of the +native ones. This provides robustness against portability issues with +these system tools. Most often the tests are written, reviewed and +tested on Linux system so such portability issues arise from time to +time. + + Running Tests ------------- The easiest way to run tests is to say "make test", (or simply run the notmuch-test script). Either command will run all available tests. Alternately, you can run a specific subset of tests by simply invoking -one of the executable scripts in this directory, (such as ./search, -./reply, etc). Note that you will probably want "make test-binaries" +one of the executable scripts in this directory, (such as ./T*-search.sh, +./T*-reply.sh, etc). Note that you will probably want "make test-binaries" before running individual tests. The following command-line options are available when running tests: @@ -64,6 +89,14 @@ The following command-line options are available when running tests: Pointing this argument at a tmpfs filesystem can improve the speed of the test suite for some users. +Certain tests require precomputed databases to complete. You can fetch these +databases with + + make download-test-databases + +If you do not download the test databases, the relevant tests will be +skipped. + When invoking the test suite via "make test" any of the above options can be specified as follows: @@ -72,9 +105,22 @@ can be specified as follows: You can choose an emacs binary (and corresponding emacsclient) to run the tests in one of the following ways. - TEST_EMACS=my-special-emacs TEST_EMACSCLIENT=my-emacsclient make test - TEST_EMACS=my-special-emacs TEST_EMACSCLIENT=my-emacsclient ./emacs - make test TEST_EMACS=my-special-emacs TEST_EMACSCLIENT=my-emacsclient + TEST_EMACS=my-emacs TEST_EMACSCLIENT=my-emacsclient make test + TEST_EMACS=my-emacs TEST_EMACSCLIENT=my-emacsclient ./T*-emacs.sh + make test TEST_EMACS=my-emacs TEST_EMACSCLIENT=my-emacsclient + +Some tests may require a c compiler. You can choose the name and flags similarly +to with emacs, e.g. + + make test TEST_CC=gcc TEST_CFLAGS="-g -O2" + +Quiet Execution +--------------- + +Normally, when new script starts and when test PASSes you get a message +printed on screen. This printing can be disabled by setting the +NOTMUCH_TEST_QUIET variable to a non-null value. Message on test +failures and skips are still printed. Skipping Tests -------------- @@ -96,16 +142,23 @@ Note that some tests in the existing test suite rely on previous test items, so you cannot arbitrarily skip any test and expect the remaining tests to be unaffected. +Currently we do not consider skipped tests as build failures. For +maximum robustness, when setting up automated build processes, you +should explicitly skip tests, rather than relying on notmuch's +detection of missing prerequisites. In the future we may treat tests +unable to run because of missing prerequisites, but not explicitly +skipped by the user, as failures. + Writing Tests ------------- -The test script is written as a shell script. It should start with -the standard "#!/usr/bin/env bash" with copyright notices, and an -assignment to variable 'test_description', like this: +The test script is written as a shell script. It is to be named as +Tddd-testname.sh where 'ddd' is three digits and 'testname' the "bare" +name of your test. Tests will be run in order the 'ddd' part determines. + +The test script should start with the standard "#!/usr/bin/env bash" +and an assignment to variable 'test_description', like this: #!/usr/bin/env bash - # - # Copyright (c) 2005 Junio C Hamano - # test_description='xxx test (option --frotz) @@ -117,7 +170,7 @@ Source 'test-lib.sh' After assigning test_description, the test script should source test-lib.sh like this: - . ./test-lib.sh + . ./test-lib.sh || exit 1 This test harness library does the following things: @@ -147,16 +200,21 @@ Test harness library There are a handful helper functions defined in the test harness library for your script to use. - test_expect_success