Austin Clements [Sun, 16 Dec 2012 03:17:23 +0000 (22:17 -0500)]
cli: Framework for structured output versioning
Currently there is a period of pain whenever we make
backward-incompatible changes to the structured output format, which
discourages not only backward-incompatible improvements to the format,
but also backwards-compatible additions that may not be "perfect". In
the end, these problems limit experimentation and innovation.
This series of patches introduces a way for CLI callers to request a
specific format version on the command line and to determine if the
CLI does not supported the requested version (and perhaps present a
useful diagnostic to the user). Since the caller requests a format
version, it's also possible for the CLI to support multiple
incompatible versions simultaneously, unlike the alternate approach of
including version information in the output.
This patch lays the groundwork by introducing a versioning convention,
standard exit codes, and a utility function to check the requested
version and produce standardized diagnostic messages and exit
statuses.
Austin Clements [Sat, 15 Dec 2012 20:04:19 +0000 (15:04 -0500)]
emacs: Use unified error handling in search
This slightly changes the output of an existing test since we now
report non-zero exits with a pop-up buffer instead of at the end of
the search results.
Austin Clements [Sat, 15 Dec 2012 20:04:17 +0000 (15:04 -0500)]
emacs: Improve error handling for notmuch-call-notmuch-json
This checks for non-zero exit status from JSON CLI calls and pops up
an error buffer with stderr and stdout. A consequence of this is that
show and reply now handle errors, rather than ignoring them.
Austin Clements [Sat, 15 Dec 2012 20:04:14 +0000 (15:04 -0500)]
emacs: Centralize notmuch command error handling
This provides library functions for unified handling of errors from
the notmuch CLI. Follow-up patches will convert some scattered error
handling to use this and add error handling where we currently ignore
errors.
Mark Walters [Sat, 8 Dec 2012 12:41:34 +0000 (12:41 +0000)]
contrib: pick: archive message updated
Update pick's archive message to respect notmuch-archive-tags. Also
split archive message into an archiving part and a separate
"then-next" part, to move more inline with show. Update the keybinding
so default behaviour is unchanged.
Austin Clements [Fri, 14 Dec 2012 15:07:35 +0000 (10:07 -0500)]
emacs: Fix bug in resynchronizing after a JSON parse error
Previously, if the input stream consisted only of an error message,
notmuch-json-begin-compound would signal a (wrong-type-argument
number-or-marker-p nil) error when reaching the end of the error
message. This happened because notmuch-json-scan-to-value would think
that it reached a value and put the parser into the 'value state.
Even after notmuch-json-begin-compound signaled the syntax error, the
parser would remain in this state and when the resynchronization logic
reached the end of the buffer, the parser would fail because the
'value state indicates that characters are available.
This fixes this problem by restoring the parser's previous state if it
encounters a syntax error.
David Bremner [Thu, 6 Dec 2012 02:12:16 +0000 (22:12 -0400)]
perf-test: use nmbug tags in dump-restore tests
This makes the tag set a bit less trivial.
Note that if you use the small corpus, this is not so interesting (and
is also a bit noisy) since the messages will not be found. In the
future this could be checked for.
David Bremner [Tue, 4 Dec 2012 02:51:14 +0000 (22:51 -0400)]
perf-test: add corpus size to output, compact I/O stats
Austin suggested a while ago that the corpus size be printed in the
header. In the end it seems the corpus will be fixed per test script,
so this suggestion indeed makes sense.
The tabbing was wrapping on my usual 80 column terminal, so I joined
the input and output columns together.
David Bremner [Thu, 6 Dec 2012 01:37:29 +0000 (21:37 -0400)]
perf-test: propagate non-zero returns from /usr/bin/time
Unlike in the correctness tests, the most common cause of non-zero
return seems to be the user interrupting, so killing the run seems
like the friendly thing to do.
Pieter Praet [Fri, 24 Feb 2012 22:30:35 +0000 (23:30 +0100)]
test: emacs: new tests "notmuch-show: {add,remove} multiple tags {to,from} single message"
* test/emacs:
- Rename subtests "{Add,Remove} tag from notmuch-show view" to
"notmuch-show: {add,remove} single tag {to,from} single message"
to be consistent with the following tests.
- New subtest "notmuch-show: add multiple tags to single message":
`notmuch-show-add-tag' ("+") can add multiple tags to a message.
- New subtest "notmuch-show: remove multiple tags from single message":
`notmuch-show-remove-tag' ("-") can remove multiple tags from a message.
Mark Walters [Sun, 9 Dec 2012 20:17:29 +0000 (20:17 +0000)]
contrib: pick: bugfix for pick splitting the window excessively
Previously if you carried on past the last message in a pick view pick
would get confused and `forget' about the split pane and would try and
re-split when moving up again. This was due to faulty logic in
notmuch-pick-show-message: something that should have been in the (when message)
clause was not.
Tomi Ollila [Sun, 9 Dec 2012 11:56:35 +0000 (13:56 +0200)]
NEWS: under-the-hood Emacs interface fixes
Added the following Emacs Interface NEWS entries:
Catch errors bodypart insertions may throw,
Improved text/calendar content handling and
Disabled coding conversions when reading in
`with-current-notmuch-show-message`.
Thanks to Austin and David for content improvements.
David Bremner [Tue, 13 Nov 2012 04:23:49 +0000 (00:23 -0400)]
tag-util: optimization of tag application
The idea is not to bother with restore operations if they don't change
the set of tags. This is actually a relatively common case.
In order to avoid fancy datastructures, this method is quadratic in
the number of tags; at least on my mail database this doesn't seem to
be a big problem.
David Bremner [Sat, 8 Dec 2012 12:34:00 +0000 (08:34 -0400)]
test/dump-restore: add test for warning/error messages
We want to test both that error/warning messages are generated when
they should be, and not generated when they should not be. This varies
between restore and batch tagging.
David Bremner [Sun, 11 Nov 2012 14:13:48 +0000 (10:13 -0400)]
notmuch-restore: add support for input format 'batch-tag'
This can be enabled with the new --format=batch-tag command line
option to "notmuch restore". The input must consist of lines of the
format:
+<tag>|-<tag> [...] [--] id:<msg-id>
Each line is interpreted similarly to "notmuch tag" command line
arguments. The delimiter is one or more spaces ' '. Any characters in
<tag> and <search-terms> MAY be hex encoded with %NN where NN is the
hexadecimal value of the character. Any ' ' and '%' characters in
<tag> and <msg-id> MUST be hex encoded (using %20 and %25,
respectively). Any characters that are not part of <tag> or
<search-terms> MUST NOT be hex encoded.
Leading and trailing space ' ' is ignored. Empty lines and lines
beginning with '#' are ignored.
Commit message mainly stolen from Jani's batch tagging commit, to
follow.
David Bremner [Tue, 13 Nov 2012 01:54:16 +0000 (21:54 -0400)]
tag-util.[ch]: New files for common tagging routines
These are meant to be shared between notmuch-tag and notmuch-restore.
The bulk of the routines implement a "tag operation list" abstract
data type act as a structured representation of a set of tag
operations (typically coming from a single tag command or line of
input).
David Bremner [Sat, 24 Nov 2012 13:43:42 +0000 (09:43 -0400)]
util: add string-util.[ch]
This is to give a home to strtok_len. It's a bit silly to add a header
for one routine, but it needs to be shared between several compilation
units (or at least that's the most natural design).
David Bremner [Thu, 14 Jun 2012 22:08:42 +0000 (01:08 +0300)]
notmuch-dump: add --format=(batch-tag|sup)
sup is the old format, and remains the default, at least until
restore is converted to parse this format.
Each line of the batch-tag format is modelled on the syntax of notmuch tag:
- "notmuch tag" is omitted from the front of the line
- The dump format only uses query strings of a single message-id.
- Each space seperated tag/message-id is 'hex-encoded' to remove
trouble-making characters.
- It is permitted (and will be useful) for there to be no tags before
the query.
In particular this format won't have the same problem with e.g. spaces
in message-ids or tags; they will be round-trip-able.
Peter Feigl [Thu, 6 Dec 2012 21:12:12 +0000 (22:12 +0100)]
Rename the -json printer functions in notmuch-reply and notmuch-show to generic -sprinter functions.
All the structured output functions in notmuch-reply and notmuch-show
are renamed to a generic name (as they do not contain any json-specific
code anyway). This patch is a preparation to actually using the new
S-Expression sprinter in notmuch-reply and notmuch-show.
Peter Feigl [Thu, 6 Dec 2012 21:12:11 +0000 (22:12 +0100)]
Adding an S-expression structured output printer.
This commit adds a structured output printer for Lisp
S-Expressions. Later commits will use this printer in notmuch search,
show and reply.
The structure is the same as json, but:
- arrays are written as lists: ("foo" "bar" "baaz" 1 2 3)
- maps are written as p-lists: (:key "value" :other-key "other-value")
- true is written as t
- false is written as nil
- null is written as nil
Jani Nikula [Tue, 4 Dec 2012 21:26:31 +0000 (23:26 +0200)]
test: fix count test
The quoting for ${SEARCH} is broken when it's supposed to be '*', and
it seems tricky to get it right. Just drop the variable and use '*'
directly. Before this, none of the messages ever matched, and the test
was comparing zeros.
Austin Clements [Tue, 4 Dec 2012 15:24:14 +0000 (10:24 -0500)]
test: Fix UTF-8 JSON tests in Python 3
test_expect_equal_json uses json.tool from the system Python. While
Python 2 wasn't picky about the encoding of stdin, Python 3 decodes
stdin strictly according to the environment. Since we set LC_ALL=C
for the tests, Python 3's json.tool was assuming stdin would be in
ASCII and aborting when it couldn't decode the UTF-8 characters from
some of the JSON tests. This patch sets the PYTHONIOENCODING
environment variable to utf-8 when invoking json.tool to override
Python's default encoding choice.
Peter Feigl [Tue, 4 Dec 2012 13:47:00 +0000 (14:47 +0100)]
Changing build tool for test/random-corpus to CXX instead of CC.
Without this change, GCC complains as follows:
gcc test/random-corpus.o test/database-test.o notmuch-config.o command-line-arguments.o lib/libnotmuch.a util/libutil.a parse-time-string/libparse-time-string.a -o test/random-corpus -lgmime-2.6 -lgio-2.0 -lgobject-2.0 -lglib-2.0 -Wl,-rpath,/usr/lib -ltalloc -lxapian
/usr/bin/ld: lib/libnotmuch.a(database.o): undefined reference to symbol '_ZNSs4_Rep10_M_destroyERKSaIcE@@GLIBCXX_3.4'
/usr/bin/ld: note: '_ZNSs4_Rep10_M_destroyERKSaIcE@@GLIBCXX_3.4' is defined in DSO /usr/lib/libstdc++.so.6 so try adding it to the linker command line
/usr/lib/libstdc++.so.6: could not read symbols: Invalid operation
collect2: error: ld returned 1 exit status
make: *** [test/random-corpus] Error 1
Mark Walters [Wed, 5 Dec 2012 12:11:49 +0000 (12:11 +0000)]
emacs: show: refresh buffer did not remove overlays
Previously refreshing the notmuch show buffer did not remove overlays
which meant that if the user refreshed a message with images the
images would remain and then the new text was added after.
One might have guessed that erase-buffer would have removed them but
it seems not. Thus force the removal of overlays with remove-overlays.
Jani Nikula [Mon, 3 Dec 2012 20:56:13 +0000 (22:56 +0200)]
cli: convert "notmuch new" to the argument parser
Use the notmuch argument parser to handle arguments in "notmuch
new". As a side effect, this fixes broken STRNCMP_LITERAL usage that
accepts, for example, --verbosefoo for --verbose.
David Bremner [Sun, 5 Aug 2012 18:13:13 +0000 (15:13 -0300)]
test: add broken roundtrip test
We demonstrate the current notmuch restore parser being confused by
message-id's and tags containing non alpha numeric characters
(particularly space and parentheses are problematic because they are
not escaped by notmuch dump).
We save the files as hex escaped on disk so that terminal emulators
will not get confused if the test fails (as we mostly expect it to do).
David Bremner [Sun, 5 Aug 2012 18:13:11 +0000 (15:13 -0300)]
test: add database routines for testing
Initially, provide a way to create "stub" messages in the notmuch
database without corresponding files. This is essentially cut and
paste from lib/database.cc. This is a seperate file since we don't
want to export these symbols from libnotmuch or bloat the library with
non-exported code.
Mark Walters [Tue, 31 Jul 2012 16:29:39 +0000 (17:29 +0100)]
emacs: Split the function notmuch-show-clean-address
This function is also used by pick so split it out in preperation for
moving to lib. In fact, pick and show want a slightly different
combination of name and email on return so make the separated function
return them as a pair, and let show or pick extract the combination
they want from that.
David Bremner [Wed, 28 Nov 2012 12:25:52 +0000 (08:25 -0400)]
configure: really expand libdir_expanded
It turns out that if people really use configure in autotools style and pass
libdir containing '${prefix}/foo' then the ldconfig previously failed.
This uses sed for portability (versus bash parameter expansion with
substitution) and hopefully a bit more robustness than blindly
parameter expanding the string.
Austin Clements [Wed, 28 Nov 2012 04:54:01 +0000 (23:54 -0500)]
test: Use associative arrays to track external prereqs
Previously, the test framework generated a variable name for each
external prereq as a poor man's associative array. Unfortunately,
prereqs names may not be legal variable names, leading to
unintelligible bash errors like
test_missing_external_prereq_emacsclient.emacs24_=t: command not found
Using proper associative arrays to track prereqs, in addition to being
much cleaner than generating variable names and using grep to
carefully construct unique string lists, removes restrictions on
prereq names.
Austin Clements [Wed, 28 Nov 2012 04:13:16 +0000 (23:13 -0500)]
test: Abort driver if a test script aborts
Previously, if a test script aborted (e.g., because it passed too few
arguments to a test function), the test driver loop would simply
continue on to the next test script and the final results would
declare that everything passed (except that the test count would look
suspiciously low, but maybe you just misremembered how many tests
there were).
Now, if a test script exits with a non-zero status and did not produce
a final results file, we propagate that failure out of the driver loop
immediately.
To keep this simple, this patch removes the PID from the test-results
file name. This PID was inherited from the git test system and seems
unnecessary, since the file name already includes the name of the test
script and the test-results directory is created anew for each run.
Austin Clements [Wed, 28 Nov 2012 03:24:59 +0000 (22:24 -0500)]
test: Make the emacsclient binary user-configurable
And require that if TEST_EMACS is specified, so is TEST_EMACSCLIENT.
Previously, the test framework always used "emacsclient", even if the
Emacs in use was overridden by TEST_EMACS. This causes problems if
both Emacs 23 and Emacs 24 are installed, the Emacs 23 emacsclient is
the system default, but TEST_EMACS is set to emacs24. Specifically,
with an Emacs 24 server and an Emacs 23 client, emacs tests that run
very quickly may produce no output from emacsclient, causing the test
to fail.
The Emacs server uses a very simple line-oriented protocol in which
the client sends a request to evaluate an expression and the server
sends a request to print the result of evaluation. Prior to Emacs bzr
commit 107565 on March 11th, 2012 (released in Emacs 24.1), if
multiple commands were sent to the emacsclient between when it sent
the evaluation command and when it entered its receive loop, it would
only process the first response command, ignoring the rest of the
received buffer. This wasn't a problem with the Emacs 23 server
because it sent only the command to print the evaluation result.
However, the Emacs 24 server first sends an unprompted command
specifying the PID of the Emacs server, then processes the evaluation
request, then sends the command to print the result. If the
evaluation is fast enough, it can send both of these commands before
emacsclient enters the receive loop. Hence, if an Emacs 24 server is
used with an Emacs 23 emacsclient, it may miss the response printing
command, ultimately causing intermittent notmuch test failures.
Austin Clements [Tue, 27 Nov 2012 04:58:17 +0000 (23:58 -0500)]
test: Quote $output in calls to test_expect_equal
Previously, many tests in emacs-subject-to-filename didn't quote the
$output argument to test_expect_equal. As a result, if $output was
empty, test_expect_equal would be passed only one argument and would
abort the entire test script. By quoting the argument, we ensure
test_expect_equal will always receive two arguments.
Austin Clements [Sun, 25 Nov 2012 05:25:45 +0000 (00:25 -0500)]
new: Skip ignored broken symlinks
We now test for user ignore patterns before attempting to determine if
a directory entry is itself a directory. As a result, we no longer
abort for broken symlinks if the user has explicitly ignored them.
This fixes the test added in the previous patch. It also slightly
changes the debug output checked by another test of ignores.
Tomi Ollila [Sun, 18 Nov 2012 20:06:25 +0000 (22:06 +0200)]
emacs: less guessing of character set in messages
The macro with-current-notmuch-show-message executes command
`notmuch show --format=raw id:...` which just outputs the contents
of the mail file verbatim (into temporary buffer). In case e.g. utf-8
locale is used the temporary buffer has buffer-file-coding-system as
utf-8. In this case Emacs converts the data to multibyte format, guessing
that input is in utf-8.
However, the "raw" (MIME) message may contain octet data in any other
8bit format, and as no (MIME-)content spesific handling to the message
is done at this point, conversion to other formats may lose information.
By setting coding-system-for-read 'no-conversion drops the conversion part
and makes this handle input as notmuch-get-bodypart-internal() does.
This marks the broken test in previous change fixed.
Austin Clements [Sun, 25 Nov 2012 06:16:01 +0000 (01:16 -0500)]
lib: Reject multi-message mboxes and deprecate single-message mbox
Previously, we would treat multi-message mboxes as one giant email,
which, besides the obvious incorrect indexing, often led to
out-of-memory errors for archival mboxes. Now we explicitly reject
multi-message mboxes. For historical reasons, we retain support for
single-message mboxes, but official deprecate this behavior.
Austin Clements [Sun, 25 Nov 2012 06:16:00 +0000 (01:16 -0500)]
test: Test for ignoring multi-message mbox
This test is currently broken. Note that its brokenness cascades and
causes the next test to fail as well (because notmuch incorrectly
indexes the mbox file).
David Bremner [Sat, 17 Nov 2012 16:28:15 +0000 (12:28 -0400)]
test: initial performance testing infrastructure
This is not near as fancy as as the unit tests, on the theory that
the code should typically be crashing when performance tuning.
Nonetheless, there is plenty of room for improvement. Several more of
the pieces of the test infrastructure (e.g. the option parsing) could
be factored out into test/test-lib-common.sh
Tomi Ollila [Thu, 8 Nov 2012 12:08:00 +0000 (14:08 +0200)]
notmuch-show.el: handle the case where icalendar-import-buffer returns nil
icalendar-import-buffer can fail by an error signal (which have been
witnessed) but according to its docstring it can also return nil
when failing (it returns t when succeeding).
Now that the error is caught by the caller of notmuch-show-inset-part-*
functions in case icalendar-import-buffer returns nil an explicit
error is signaled and unwind-protect takes care of deleting the
temporary file (just in case, it is usually not written to the fs yet).
Tomi Ollila [Thu, 8 Nov 2012 12:07:59 +0000 (14:07 +0200)]
notmuch-show.el: import calendar data with public function after CR removal
notmuch-get-bodypart-content provides raw data to its caller so
that it can be stored verbatim whenever needed. icalendar functions
expect Emacs to do EOL conversion for the data given to these. Therefore
it the CRLF -> LF conversion is now done explicitly.
The calls to private functions icalendar--convert-ical-to-diary and
icalendar--read-element are replaced with call to public function
icalendar-import-buffer.
Austin Clements [Tue, 20 Nov 2012 15:58:45 +0000 (10:58 -0500)]
test: Produce useful output when the HTML with images test fails
Previously, this would simply indicate that the grep failed without
any indication of the Emacs output it failed on. Now we take
advantage of the test framework's handling of stdout to display the
incorrect Emacs output if the test fails.
Mark Walters [Wed, 7 Nov 2012 20:43:27 +0000 (20:43 +0000)]
contrib: notmuch-pick: add tests
The test should be run using the wrapper run-tests.sh. This links
the tests into the normal notmuch TEST_DIRECTORY and runs them from
there. After the test is complete then the links are removed.
David Bremner [Fri, 16 Nov 2012 12:31:19 +0000 (08:31 -0400)]
test: add nontrivial test for restore --accumulate.
It seems we have never tested the case that restore --accumulate
actually adds tags. I noticed this when I started optimizing and no
tests failed.
The bracketing with "restore --input=dump.expected" are to make sure
we start in a known state, and we leave the database in a known state
for the next test.
Mark Walters [Sat, 10 Nov 2012 17:02:18 +0000 (17:02 +0000)]
contrib: pick: bugfix when trying to show a non-message
If the user pressed return on the end result status line it gave a
blank message. Modify the function notmuch-pick-get-message-id to
return nil rather than an empty message-id in this case to fix this.
This also fixes a bug in the (lack of) quoting of the id string.
David Bremner [Thu, 15 Nov 2012 01:33:21 +0000 (21:33 -0400)]
notmuch-restore: tidy formatting
This is again the work of uncrustify.
I remember there is some controversy about "! foo" versus "!foo", but
in context I think "! foo" looks OK. Also, for functions "! foo
(blah)" seems better than "!foo (blah)".
Tomi Ollila [Fri, 16 Nov 2012 10:33:03 +0000 (12:33 +0200)]
NEWS (for 0.15): Added note about Bcc & Reply-To headers in json output
notmuch show --format json now includes Bcc and Reply-To headers of
the message. Mention that in NEWS.
(Heavily modified version of text originally from Michal Nazarewicz.)