notmuch/test/test-lib.sh, branch 0.9

test: Report test failures from test_expect_*

2011-09-13T02:36:00Z

This makes test_expect_* return non-zero if the test fails, so the caller can make decisions based on this, such as setting test prerequisites.

test: Fix message when skipping test_expect_equal* tests

2011-09-13T02:35:59Z

For these types of tests, the test name is previously recorded in a variable, not passed to the test function, so pass this variable to test_skip.

test: reset test_subtest_known_broken_ after each success/failure.

2011-09-13T02:35:21Z

This means that test_subtest_known_broken needs to be called before every known broken subtest, which is no different than what is documented for the test_begin_subtest case. The assumption is that every test ends up calling either skipping, calling test_ok_ or test_failure_ and and the latter in turn delegate to the known_broken versions in the case where test_subtest_known_broken_ is set.

test: improve known broken tests support

2011-09-10T13:13:27Z

There is existing support for broken tests. But it is not convenient to use. The primary issue is that we have to maintain a set of test_expect_*_failure functions which are equivalent to the normal test_expect_* counterparts except for what functions are called for result reporting. The patch adds test_subtest_known_broken function which marks a subset as broken, making the normal test_expect_* functions behave as test_expect_*_failure. All test_expect_*_failure functions are removed. Test_known_broken_failure_ is changed to format details the same way as test_failure_ does. Another benefit of this change is that the diff when a broken test is fixed would be small and nice. Documentation is updated accordingly.

test: Nix increment_mtime.

2011-06-29T22:26:45Z

With the fix for the mtime race, this workaround is no longer necessary.

test: use emacsclient(1) for Emacs tests

2011-06-29T00:10:55Z

Before the change, every Emacs test ran in a separate Emacs instance. Starting Emacs many times wastes considerable time and it gets worse as the test suite grows. The patch solves this by using a single Emacs server and emacsclient(1) to run multiple tests. Emacs server is started on the first test_emacs call and stopped when test_done is called. We take care not to leave orphan Emacs processes behind when test is terminated by whatever reason: Emacs server runs a watchdog that periodically checks that the test is still running. Some tests need to provide user input. Before the change, this was done using echo(1) to Emacs stdin. This no longer works and instead `standard-input' variable is set accordingly to make `read' return the appropriate string.

test: Increment mtime when delivering a message with emacs_deliver_message

2011-06-29T00:10:55Z

Without this, mail messages delivered by emacs_deliver_message might not be seen by the next invocation of "notmuch new", (which can lead to test-suite failures if emacs_deliver_message is fast enough).

test:Improve test behaviors when --root is used

2011-06-28T23:01:56Z

Change add_email_corpus, emacs_deliver_message and tests to use $TEST_DIRECTORY instead of '..'. This improves the behavior of the usage of --root=, as the assumption of what '..' means will usually be incorrect. Document -root option in README and update valgrind to work with -root.

test: generate run_emacs script once on test startup

2011-06-28T22:06:47Z

Instead of generating auxiliary run_emacs script every time test_emacs is run, do it once in the beginning of the test. Also, use absolute paths in the script to make it more robust.

test: set variables using `let' instead of `setq' in Emacs tests

2011-06-28T22:06:47Z

Using `setq' for setting variables in Emacs tests affect other tests that may run in the same Emacs environment. Currently it works because each test is run in a separate Emacs instance. But in the future multiple tests will run in a single Emacs instance. The patch changes all variables to use `let', so the scope of the change is limited to a single test.