notmuch/test, branch 0.2

make test: Actually count and report on failures.

2010-04-15T22:18:30Z

Hurrah---no more manual verification of that PASS column. This means that "make test" can actually be a useful part of the release process now, (since it will exit with non-zero status if there are any failures).

test: Unify all tests to use the pass_if_equal function.

2010-04-15T22:09:21Z

Previously some tests (dump/restore) were doing ad-hoc verification of values and their own printing of PASS/FAIL, etc. This made it impossible to count test pass/fail rates in a single place. The only reason these tests were written that way was because the old execute_expecting function only worked if one could directly test the stdout output of a notmuch command. The recent switch to pass_if_equal means that all tests can use it.

test: Align the PASS/FAIL column.

2010-04-15T22:06:00Z

Just to make things a little more pleasant to look at.

test: Add a test for "notmuch search '*'"

2010-04-15T21:48:56Z

This feature was added recently and should have gotten a new test at the time. As this test demonstrates, the code is broken, ("notmuch search '*' returns bogus dates of the Unix epoch for any threads where the term "and" does not appear in any messages).

test: Use an older date for the generated messages.

2010-04-15T21:13:12Z

Using a date in the current year makes the test suite fragile since the search output will include a date of "January 05" for now, but will start doing "2010-01-05" in the future.

test: Remove test-message filenames from generated messages

2010-04-15T21:13:12Z

The filenames aren't predictable (including the current directory) nor stable from one run to the next (including the PID). This makes it hard to predict the output from a search command that returns such a message (such as "*"). The original goal was simply to ensure that each generated message was distinguishable somehow. So just use the message counter instead.

test: Disentangle execution of notmuch from the examination of the results.

2010-04-15T21:13:12Z

The old execute_expecting function was doing far too much for its own good. One of the worst aspects of this was that it introduced shell-quoting challengers where the caller could not easily control the precise invocation of the command to be executed. I personally couldn't find a way to test "notmuch search '*'" without the shell expanding * against files in the current directory, or having bogus quotation marks appearing in the search string, (defeating the recognition of "*" as a special search term). Hopefully this aspect of the test suite will be much easier to maintain now.

test: Fix reply tests to track insignificant change in output.

2010-04-15T00:17:50Z

The recent fix to properly decode encoded headers made the expected output of "notmuch reply" differ by a single space, (previously, there were two spaces before the References: value and now there is just one). Fix the test suite so that these are all noted as correct results again.

test: Add new tests for out-of-order messages.

2010-04-12T21:04:35Z

These new tests demonstrate a bug as follows: Multiple messages are added to the database All of these message references a common parent The parent message does not exist in the databas In this scenario, the messages will not be recognized as belonging to the same thread. We consider this a bug, and the new tests treat this as a failure. Edited by Carl Worth : Split these tests into their own commit (before the fix of the bug). This lets me see the actual failure in the test suite, before the fix is applied. Also fix the alignment of new messages from test suite, (so that the PASS portions all line up---which is important while we're still manually verifying test-suite results).

test: Fix phrase-search tests.

2010-03-31T20:31:09Z

With some extra qutotation marks, we are now doing actual phrase searches so these tests pass.