test: support testing notmuch as installed We put some effort into testing the built copy rather than some installed copy. On the other hand for people like packagers, testing the installed copy is also of interest. When NOTMUCH_TEST_INSTALLED is set to a nonempty value, tests do not require a built notmuch tree or running configure. Some of the tests marked as broken when running against installed notmuch are probably fixable.
test: use --minversion to detect GNU Parallel. Based on a suggestion of Ole Tange [1]. [1]: id:CA+4vN7x6jp4HCiEybZ=5g+2X6Pa7etBFX3Bbd=UYty37gR6wEQ@mail.gmail.com
test: aggregate-results updates notmuch-test will now call aggregate-results.sh with file list that it compiles based on the test ran, and aggregate-results will report failure is any of the test files are missing. With this notmuch-test no longer has to exit in non-parallel run if some test fail to write its report file -- so it works as parallel tests in this sense. Changed test_done() in test-lib.sh write report file in one write(2), so there is (even) less chance it being partially written. Also, now it writes 'total' last and aggregate-results.sh expects this line to exist in all report files for reporting to be successful. Added 'set -eu' to notmuch-test and modified code to work with these settings. That makes it harder to get mistakes slipped into committed code.
test: hide message from moreutils parallel. The argument --version confuses moreutils parallel, but this is OK, because its confusing does not include printing "GNU"
test: check for GNU parallel with --version options The lastest versions of GNU parallel no longer make mention of GNU within their help output. This causes the test script to mistakenly use the moreutils parallel execution. In order to fix this, while maintaining compatibility with previous versions of GNU parallel, --version should be used. Signed-off-by: Tobias Backer Dirks <omgitsaheadcrab@gmail.com>
test: report summary even when aborting In certain cases of test suite failure, the summary report was not being printed. In particular, any failure on the parallel test suite, and any aborted test in the serialized test suite would end up hiding the summary. It's better to always show the summary where we can (while preserving the return code). If we do abort due to this high-level failure, though, we should also announce to the user that we're doing so as close to the end of the process as possible, to make it easier to find the problem. Signed-off-by: Daniel Kahn Gillmor <dkg@fifthhorseman.net>
test: allow disabling timeout with NOTMUCH_TEST_TIMEOUT=0 To aid in diagnosing test suite tooling that interacts poorly with coreutils' timeout, it's handy to be able to bypass it entirely. Signed-off-by: Daniel Kahn Gillmor <dkg@fifthhorseman.net>
test-lib.sh: colors to test output when parallel(1) is run on tty Done via $COLORS_WITHOUT_TTY environment variable as passing options to commands through parallel(1) does not look trivial. Reorganized color checking in test-lib.sh a bit for this (perhaps were not fully necessary but rest still an improvement): - color checking commands in subshell are not run before arg parsing (args may disable colors with --no-color) - [ -t 1 ] is checked before forking subshell
tests: fail and report when a parallel build fails (or times out) When a parallel build fails (or when it times out, if timeout is present), the test suite should not blithely succeed. Catch these failures and at least report them. Signed-off-by: Daniel Kahn Gillmor <dkg@fifthhorseman.net>
tests: make timeout configurable with NOTMUCH_TEST_TIMEOUT (default: 2m) The current 2 minute timeout is reasonable, but to exercise the test suite or induce timeout failures, we might want to make it shorter. This makes it configurable so you can run (for example): make check NOTMUCH_TEST_TIMEOUT=10s We stick with the default of 2m. Signed-off-by: Daniel Kahn Gillmor <dkg@fifthhorseman.net>
tests: environment variable to specify that tests should be serialized If NOTMUCH_TEST_SERIALIZE is non-null all tests will be run in series, rather than in parallel.
tests: run all tests in parallel, if available If either the moreutils or GNU parallel utility are available, run all tests in parallel. On my eight core machine this makes for a ~x7 speed-up in the full test suite (1m24s -> 12s). The design of the test suite makes this parallelization trivial.
tests: remove entangling corpus index optimization The add_email_corpus test utility includes logic that tries to re-use an index of the corpus if available. This was seemingly done as an optimization, so that every test that uses the corpus didn't have to create it's own index of the corpus. However, this has the perverse side effect of entangling tests together, and breaks parallelization. Forcing each test to do it's own index does increase the overall time of the test slightly (~6%), but this will be more than made up for in the next patch that introduces paraellization.
test: use source and build paths in notmuch-test Make a distinction between source and build directories.
test: Update extant references to corpus.mail 971cdc72cdb80f060193bc0914dc9badcc29696b renamed corpus.mail to corpora.mail. Although 971cdc72cdb80f060193bc0914dc9badcc29696b updated some of the remaining corpus.mail references, two remained, causing the test suite to leave behind an unignored corpora.mail directory.
test/notmuch-test: simple streamlining A few simple things that applies well to test/notmuch-test.sh - Shell does pathname expansion also without doing `echo ...` in subshell. - Redirections >/dev/null 2>/dev/null hide (improbable but) potential serious errors; adding -f to rm instead. - Inter-file capitalization consistency in comments. - Unnecesary space removal.
test: copyright information updates Files in test directories had only copyright of a single individual, of which code was adapted here as a base of the test system. Since then many Notmuch Developers have contributed to the test system, which is now acknowledged with a constant string in some of the test files. The README file in test directory instructed new files contain a copyright notice, but that has never been done (and it is also not needed). To simplify things a bit (and lessen confusion) this instruction is now removed. As a side enchangement, all of the 3 entries in the whole source tree cd'ing to `dirname` of "$0" now uses syntax cd "$(dirname "$0")". This makes these particular lines work when current working directory is e.g. /c/Program Files/notmuch/test/. (Probably it would fail elsewhere, though.)
test: renamed test scripts to format T\d\d\d-name.sh All test scripts to be executed are now named as T\d\d\d-name.sh, numers in increments of 10. This eases adding new tests and developers to see which are test scripts that are executed by test suite and in which order.
test: print empty line at the beginning of test script, not at end In preparation for quiet mode print empty line before writing the test description. This is done now in function designed for it -- it will also be called when test fails.
test: move emacs-tree test into mainline We move the emacs-tree test and associated files into the main test directory and add the test to the list in notmuch-test.