notmuch/test/notmuch-test, branch 0.40

test: support testing notmuch as installed

2023-07-21T10:41:50Z

We put some effort into testing the built copy rather than some installed copy. On the other hand for people like packagers, testing the installed copy is also of interest. When NOTMUCH_TEST_INSTALLED is set to a nonempty value, tests do not require a built notmuch tree or running configure. Some of the tests marked as broken when running against installed notmuch are probably fixable.

test: use --minversion to detect GNU Parallel.

2021-08-03T23:29:47Z

Based on a suggestion of Ole Tange [1]. [1]: id:CA+4vN7x6jp4HCiEybZ=5g+2X6Pa7etBFX3Bbd=UYty37gR6wEQ@mail.gmail.com

test: aggregate-results updates

2021-06-07T23:16:33Z

notmuch-test will now call aggregate-results.sh with file list that it compiles based on the test ran, and aggregate-results will report failure is any of the test files are missing. With this notmuch-test no longer has to exit in non-parallel run if some test fail to write its report file -- so it works as parallel tests in this sense. Changed test_done() in test-lib.sh write report file in one write(2), so there is (even) less chance it being partially written. Also, now it writes 'total' last and aggregate-results.sh expects this line to exist in all report files for reporting to be successful. Added 'set -eu' to notmuch-test and modified code to work with these settings. That makes it harder to get mistakes slipped into committed code.

test: hide message from moreutils parallel.

2021-04-26T12:00:11Z

The argument --version confuses moreutils parallel, but this is OK, because its confusing does not include printing "GNU"

test: check for GNU parallel with --version options

2021-04-24T23:06:41Z

The lastest versions of GNU parallel no longer make mention of GNU within their help output. This causes the test script to mistakenly use the moreutils parallel execution. In order to fix this, while maintaining compatibility with previous versions of GNU parallel, --version should be used. Signed-off-by: Tobias Backer Dirks

test: report summary even when aborting

2019-05-26T21:55:06Z

In certain cases of test suite failure, the summary report was not being printed. In particular, any failure on the parallel test suite, and any aborted test in the serialized test suite would end up hiding the summary. It's better to always show the summary where we can (while preserving the return code). If we do abort due to this high-level failure, though, we should also announce to the user that we're doing so as close to the end of the process as possible, to make it easier to find the problem. Signed-off-by: Daniel Kahn Gillmor

test: allow disabling timeout with NOTMUCH_TEST_TIMEOUT=0

2019-05-25T11:26:41Z

To aid in diagnosing test suite tooling that interacts poorly with coreutils' timeout, it's handy to be able to bypass it entirely. Signed-off-by: Daniel Kahn Gillmor

test-lib.sh: colors to test output when parallel(1) is run on tty

2019-05-23T11:00:31Z

Done via $COLORS_WITHOUT_TTY environment variable as passing options to commands through parallel(1) does not look trivial. Reorganized color checking in test-lib.sh a bit for this (perhaps were not fully necessary but rest still an improvement): - color checking commands in subshell are not run before arg parsing (args may disable colors with --no-color) - [ -t 1 ] is checked before forking subshell

tests: fail and report when a parallel build fails (or times out)

2019-05-20T17:48:56Z

When a parallel build fails (or when it times out, if timeout is present), the test suite should not blithely succeed. Catch these failures and at least report them. Signed-off-by: Daniel Kahn Gillmor

tests: make timeout configurable with NOTMUCH_TEST_TIMEOUT (default: 2m)

2019-05-20T17:48:43Z

The current 2 minute timeout is reasonable, but to exercise the test suite or induce timeout failures, we might want to make it shorter. This makes it configurable so you can run (for example): make check NOTMUCH_TEST_TIMEOUT=10s We stick with the default of 2m. Signed-off-by: Daniel Kahn Gillmor