notmuch/test/notmuch-test, branch 0.30_rc2

test: report summary even when aborting

2019-05-26T21:55:06Z

In certain cases of test suite failure, the summary report was not being printed. In particular, any failure on the parallel test suite, and any aborted test in the serialized test suite would end up hiding the summary. It's better to always show the summary where we can (while preserving the return code). If we do abort due to this high-level failure, though, we should also announce to the user that we're doing so as close to the end of the process as possible, to make it easier to find the problem. Signed-off-by: Daniel Kahn Gillmor

test: allow disabling timeout with NOTMUCH_TEST_TIMEOUT=0

2019-05-25T11:26:41Z

To aid in diagnosing test suite tooling that interacts poorly with coreutils' timeout, it's handy to be able to bypass it entirely. Signed-off-by: Daniel Kahn Gillmor

test-lib.sh: colors to test output when parallel(1) is run on tty

2019-05-23T11:00:31Z

Done via $COLORS_WITHOUT_TTY environment variable as passing options to commands through parallel(1) does not look trivial. Reorganized color checking in test-lib.sh a bit for this (perhaps were not fully necessary but rest still an improvement): - color checking commands in subshell are not run before arg parsing (args may disable colors with --no-color) - [ -t 1 ] is checked before forking subshell

tests: fail and report when a parallel build fails (or times out)

2019-05-20T17:48:56Z

When a parallel build fails (or when it times out, if timeout is present), the test suite should not blithely succeed. Catch these failures and at least report them. Signed-off-by: Daniel Kahn Gillmor

tests: make timeout configurable with NOTMUCH_TEST_TIMEOUT (default: 2m)

2019-05-20T17:48:43Z

The current 2 minute timeout is reasonable, but to exercise the test suite or induce timeout failures, we might want to make it shorter. This makes it configurable so you can run (for example): make check NOTMUCH_TEST_TIMEOUT=10s We stick with the default of 2m. Signed-off-by: Daniel Kahn Gillmor

tests: environment variable to specify that tests should be serialized

2019-05-07T09:55:31Z

If NOTMUCH_TEST_SERIALIZE is non-null all tests will be run in series, rather than in parallel.

tests: run all tests in parallel, if available

2019-05-07T09:54:09Z

If either the moreutils or GNU parallel utility are available, run all tests in parallel. On my eight core machine this makes for a ~x7 speed-up in the full test suite (1m24s -> 12s). The design of the test suite makes this parallelization trivial.

tests: remove entangling corpus index optimization

2019-05-07T09:53:57Z

The add_email_corpus test utility includes logic that tries to re-use an index of the corpus if available. This was seemingly done as an optimization, so that every test that uses the corpus didn't have to create it's own index of the corpus. However, this has the perverse side effect of entangling tests together, and breaks parallelization. Forcing each test to do it's own index does increase the overall time of the test slightly (~6%), but this will be more than made up for in the next patch that introduces paraellization.

test: use source and build paths in notmuch-test

2017-10-20T22:55:31Z

Make a distinction between source and build directories.

test: Update extant references to corpus.mail

2017-08-18T22:42:19Z

971cdc72cdb80f060193bc0914dc9badcc29696b renamed corpus.mail to corpora.mail. Although 971cdc72cdb80f060193bc0914dc9badcc29696b updated some of the remaining corpus.mail references, two remained, causing the test suite to leave behind an unignored corpora.mail directory.