notmuch/test/aggregate-results.sh, branch master

test: aggregate-results updates

2021-06-07T23:16:33Z

notmuch-test will now call aggregate-results.sh with file list that it compiles based on the test ran, and aggregate-results will report failure is any of the test files are missing. With this notmuch-test no longer has to exit in non-parallel run if some test fail to write its report file -- so it works as parallel tests in this sense. Changed test_done() in test-lib.sh write report file in one write(2), so there is (even) less chance it being partially written. Also, now it writes 'total' last and aggregate-results.sh expects this line to exist in all report files for reporting to be successful. Added 'set -eu' to notmuch-test and modified code to work with these settings. That makes it harder to get mistakes slipped into committed code.

test: aggregate-results.sh: count test files where all tests skipped

2019-07-05T15:58:23Z

Previously, when all tests were skipped on a test file, there were no indication of this in the final results aggregate-results.sh printed. Now count of the files where all tests were skipped is printed.

test: aggregate-results.sh: consistent style. zero forks.

2019-06-11T10:20:01Z

- all variables in $((...)) without leading $ - all comparisons use -gt, -eq or -ne - no -a nor -o inside [ ... ] expressions - all indentation levels using one tab Dropped unnecessary empty string check when reading results files. Replaced pluralize() which was executed in subshell with pluralize_s(). pluralize_s sets $s to 's' or '' based on value of $1. Calls to pluralize_s are done in context of current shell, so no forks to subshells executed.

test suite: don't consider skipped individual tests as failing

2015-09-25T12:18:22Z

It isn't completely clear what we want to do here, but 1) We currently don't fail if we skip a whole test file (mainly because we neglect to count those skipped tests properly). This change at least makes the two kinds of skipping consistent. 2) Automated build environments may have good reasons for building with a minimal set of prereqs, and we don't want to discourage running our test suite by breaking builds.

test: exit with nonzero value when not all tests completed successfully

2013-09-09T01:40:57Z

If any of the tests in our test system is not passing the execution of the test suite completes with nonzero exit value. It is better to rely on the exit value of the test system instead of some arbitrary strings in test output (or use both).

test: change "#!/bin/bash" to "#!/usr/bin/env bash" enhances portability

2011-05-27T21:03:28Z

Change #!/bin/bash at start of tests to "#!/usr/bin/env bash". That way systems running on bash < 4 can prepend bash >= 4 to path before running the tests.

test: Fix to actually report errors (!).

2010-09-20T21:39:40Z

A bug in the results-aggregation code was causing the test suite to report "all tests passed" even when there were failures, (as long as there were also no "broken" tests). Fix this.

test: Cleanup the test output

2010-09-17T21:08:36Z

This makes the new, git-derived test suite report results in a manner similar to the original notmuch test suite. Notable changes include: * No more initial '*' on every line * Only colorize a single word * Don't print useless test numbers * Use "PASS" in place of "ok" * Begin sentences with a capital letter * Print test descriptions for each block * Separate each block of tests with a blank line * Don't summarize counts between each block

Copy test framework from Git

2010-09-16T22:56:44Z

Git uses a simple and yet powerful test framework, written in shell. The framework is easy to use for both users and developers so I think it would help if it is used in notmuch as well. This is a copy of Git's test framework from commit b6b0afdc30e066788592ca07c9a6c6936c68cc11 in git repository. Signed-off-by: Michal Sojka