David Bremner [Tue, 7 Jun 2016 10:37:57 +0000 (07:37 -0300)]
lib: fix definition of LIBNOTMUCH_CHECK_VERSION
Fix bug reported in id:20160606124522.g2y2eazhhrwjsa4h@flatcap.org
Although the C99 standard 6.10 is a little non-obvious on this point,
the docs for e.g. gcc are unambiguous. And indeed in practice with the
extra space, this code fails
#include <stdio.h>
#define foo (x) (x+1)
int main(int argc, char **argv){
printf("%d\n",foo(1));
}
David Bremner [Tue, 28 Jun 2016 08:24:07 +0000 (10:24 +0200)]
test: don't use dump and restore in a pipeline
This has been wrong since bbbdf0478ea, but the race condition was not
previously been (often?) triggered in the tests. With the DB_RETRY_LOCK
patches, it manifests itself as a deadlock.
David Bremner [Sun, 26 Jun 2016 15:29:45 +0000 (17:29 +0200)]
lib: add built_with handling for XAPIAN_DB_RETRY_LOCK
This support will be present only if the appropriate version of xapian
is available _and_ the user did not disable the feature when
building. So there really needs to be some way for the user to check.
Istvan Marko [Sun, 26 Jun 2016 15:29:43 +0000 (17:29 +0200)]
Use the Xapian::DB_RETRY_LOCK flag when available
Xapian 1.3 has introduced the DB_RETRY_LOCK flag (Xapian bug
275). Detect it in configure and optionally use it. With this flag
commands that need the write lock will wait for their turn instead of
aborting when it's not immediately available.
David Bremner [Tue, 28 Jun 2016 08:24:07 +0000 (10:24 +0200)]
test: don't use dump and restore in a pipeline
This has been wrong since bbbdf0478ea, but the race condition was not
previously been (often?) triggered in the tests. With the DB_RETRY_LOCK
patches, it manifests itself as a deadlock.
Mark Walters [Thu, 9 Jun 2016 09:54:10 +0000 (10:54 +0100)]
emacs: show: improve handling of mark read tagging errors
Previously if a marking read tag change (i.e., removing the unread
tag) failed for some reason, such as a locked database, then no more
mark read tag changes would be attempted in that buffer.
This handles the error more gracefully. There is not much we can do
yet about dealing with the error itself, and marking read is probably
not important enough to warrant keeping a queue of pending changes or
anything.
However this commit changes it so that
- we do try and make future mark read tag changes.
- we display the tag state correctly: i.e. we don't display the tag as
deleted (no strike through)
- and since we know the tag change failed we can try to mark this
message read in the future. Indeed, since the code uses the
post-command hook we will try again on the next keypress (unless the
user has left the message).
We indicate to the user that these mark read tag changes may have
failed in the header-line.
Tomi Ollila [Wed, 25 May 2016 21:37:41 +0000 (00:37 +0300)]
test: test_python: set PYTHONPATH to the python execution environment
Place PYTHONPATH to the environment when python is executed in a way
that current shell environment is not affected. This also allows adding
the old value of PYTHONPATH to the end of the new value (otherwise it
would have been appended again and again when test_python is called).
At the same time, use -B option to avoid writing .pyc files to
bindings/python/* (which are not cleared out by distclean).
Drop the (unused) prefix code which preserved the original stdout of the
python program and opened sys.stdout to OUTPUT. In place of that there
is now note how (debug) information can be printed to original stdout.
Tomi Ollila [Wed, 25 May 2016 21:37:40 +0000 (00:37 +0300)]
test: set LD_LIBRARY_PATH early and keep its old contents
Previously LD_LIBRARY_PATH was exported (and environment changed)
in the middle of test case execution, when a function setting it
was called.
Previously the old contents of LD_LIBRARY_PATH was lost (if any)
when it was re-set and exported. In some systems the old contents of
LD_LIBRARY_PATH was needed to e.g. locate suitable gmime library.
Tomi Ollila [Sat, 28 May 2016 11:06:04 +0000 (14:06 +0300)]
notmuch-emacs-mua: escape $PWD (and cd always)
Escaping $PWD makes this work in directories like 'foo"bar'...
Cd'ing always makes the working directory to be consistent whether
--body option was used or not (when using emacsclient, but cd'ing
when using emacs does not cause any harm).
Note that documentation of `insert-file` expects programs to
call `insert-file-contents` instead. In our simple case
`insert-file` works better as it does some good checks that we'd
have to implement ourselves. Look lisp/files.el in emacs sources
for more information.
Tomi Ollila [Sat, 28 May 2016 12:39:30 +0000 (15:39 +0300)]
test: fix die() in test-lib-common.sh
In scripts that include test-lib-common.sh but not test-lib.sh
the die() implementation needs to be a bit different due to
fd redirection differences. test-lib-common.sh implements die()
only if it was not implemented already.
Tomi Ollila [Sat, 28 May 2016 12:39:29 +0000 (15:39 +0300)]
test: add function die () and have use of it in add_email_corpus ()
Added die() function to test-lib.sh with the following first use of it:
If notmuch new fails during email corpus addition the database is
most probably inexistent or broken and the added corpus would be
unusable while running single tests, giving misleading failures
("only" full 'make test' cleans out old corpus).
David Bremner [Tue, 7 Jun 2016 10:37:57 +0000 (07:37 -0300)]
lib: fix definition of LIBNOTMUCH_CHECK_VERSION
Fix bug reported in id:20160606124522.g2y2eazhhrwjsa4h@flatcap.org
Although the C99 standard 6.10 is a little non-obvious on this point,
the docs for e.g. gcc are unambiguous. And indeed in practice with the
extra space, this code fails
#include <stdio.h>
#define foo (x) (x+1)
int main(int argc, char **argv){
printf("%d\n",foo(1));
}
Setting locale environment variables (LC_* and LANG) to e.g.
en_US.utf8 works fine on Linux, and that is what locale -a
returns (in Linux). However this does not work e.g. in some *BSD
systems.
In these systems, en_US.UTF-8 works. This also works in Linux
systems (which may look like a surprising thing on the first sight(*)).
But that *UTF-8 format seems to be widely used in the Linux system:
Grep it through the files in /etc/, for example.
Easy way to test: Run the following command lines. First should
complain about setting locale failed, and second should not.
This isn't just ugly: it's confusing, because it seems to imply that
some of the prefixes in the left-hand column are somehow related to
specific other prefixes in the right-hand column.
The Definition List representation introduced by this patch should be
simpler for readers to understand, and doesn't have the warning.
Many of the external links found in the notmuch source can be resolved
using https instead of http. This changeset addresses as many as i
could find, without touching the e-mail corpus or expected outputs
found in tests.
NEWS, python: update pointer to online documentation
Currently, http://packages.python.org/notmuch/ goes through a series
of redirections and ends up pointing to readthedocs. Since we're
using readthedocs directly anyway, just point to it directly.
readthedocs are also now sensibly using a separate domain
(readthedocs.io) for their hosted documentation as distinct from their
own domain (readthedocs.org), so use the correct tld.
Tomi Ollila [Sat, 28 May 2016 17:45:31 +0000 (20:45 +0300)]
lib: whitespace cleanup
Cleaned the following whitespace in lib/* files:
lib/index.cc: 1 line: trailing whitespace
lib/database.cc 5 lines: 8 spaces at the beginning of line
lib/notmuch-private.h: 4 lines: 8 spaces at the beginning of line
lib/message.cc: 1 line: trailing whitespace
lib/sha1.c: 1 line: empty lines at the end of file
lib/query.cc: 2 lines: 8 spaces at the beginning of line
lib/gen-version-script.sh: 1 line: trailing whitespace
David Bremner [Tue, 22 Mar 2016 10:54:53 +0000 (07:54 -0300)]
lib: make a global constant for query parser flags
It's already kindof gross that this is hardcoded in two different
places. We will also need these later in field processors calling back
into the query parser.
David Bremner [Tue, 22 Mar 2016 10:54:52 +0000 (07:54 -0300)]
CLI: add notmuch-config support for named queries
Most of the infrastructure here is general, only the validation/dispatch
is hardcoded to a particular prefix.
A notable change in behaviour is that notmuch-config now opens the
database e.g. on every call to list, which fails with an error message
if the database doesn't exit yet.
David Bremner [Tue, 22 Mar 2016 10:54:48 +0000 (07:54 -0300)]
lib: config list iterators
Since xapian provides the ability to restrict the iterator to a given
prefix, we expose this ability to the user. Otherwise we mimic the other
iterator interfances in notmuch (e.g. tags.c).
David Bremner [Tue, 22 Mar 2016 10:54:47 +0000 (07:54 -0300)]
lib: provide config API
This is a thin wrapper around the Xapian metadata API. The job of this
layer is to keep the config key value pairs from colliding with other
metadata by transparently prefixing the keys, along with the usual glue
to provide a C interface.
The split of _get_config into two functions is to allow returning of the
return value with different memory ownership semantics.
Ludovic LANGE [Sun, 6 Mar 2016 11:56:28 +0000 (07:56 -0400)]
ruby: add bindings for `notmuch_database_get_all_tags`
The Ruby bindings were missing a way to get all the tags of the
database. Now you should be able to access this with the public
instance method `all_tags` of your database object.
Example of use:
notmuchdb = Notmuch::Database.new path, { :create => false,
:mode => Notmuch::MODE_READ_ONLY }
Tomi Ollila [Fri, 6 May 2016 15:41:57 +0000 (18:41 +0300)]
test: copyright information updates
Files in test directories had only copyright of a single individual,
of which code was adapted here as a base of the test system.
Since then many Notmuch Developers have contributed to the test
system, which is now acknowledged with a constant string in some
of the test files.
The README file in test directory instructed new files contain a
copyright notice, but that has never been done (and it is also not
needed). To simplify things a bit (and lessen confusion) this
instruction is now removed.
As a side enchangement, all of the 3 entries in the whole source
tree cd'ing to `dirname` of "$0" now uses syntax cd "$(dirname "$0")".
This makes these particular lines work when current working directory
is e.g. /c/Program Files/notmuch/test/.
(Probably it would fail elsewhere, though.)
Tomi Ollila [Sat, 7 May 2016 19:24:18 +0000 (22:24 +0300)]
cli: tell how many messages were precisely matched when expected 1 match
In case of notmuch reply and notmuch show --part=N it is required that
search terms match to one message. If match count was != 1, error
message "Error: search term did not match precisely one message"
was too vague to explain what happened.
By appending (matched <num> messages) to the error message it
makes the problem more understandable (e.g when <num> is '0'
user reckons the query had a typo in it).
Tomi Ollila [Fri, 13 May 2016 21:29:45 +0000 (00:29 +0300)]
configure: combine common parts of CONFIGURE_C{,XX}FLAGS
By combining the common parts of CONFIGURE_CFLAGS and CONFIGURE_CXXFLAGS
to a separate make variable and using that as part of their
definitions makes setting of these easier, DRYer and less error prone
(especially as we cannot check potential typing errors there).
Tomi Ollila [Fri, 6 May 2016 18:11:25 +0000 (21:11 +0300)]
configure: add set -u
In case of any unset variable, make ./configure exit with nonzero value;
an attempt to expand an unset variable is a bug in the script
(usually a spelling mistake) and those should not pass through
unnoticed.
David Edmondson [Sat, 30 Apr 2016 06:51:48 +0000 (07:51 +0100)]
emacs: Tell `message-mode' that outgoing messages are email.
When composing messages (including replies, etc.), indicate to
`message-mode' definitively that the message is email (as opposed to
Usenet news) rather than having it attempt to determine this for itself.
This causes `message-mode' to observe such variables as
`message-default-mail-headers', which previously happened haphazardly.
David Edmondson [Sat, 30 Apr 2016 06:51:47 +0000 (07:51 +0100)]
emacs: Observe the charset of MIME parts when reading them.
`notmuch--get-bodypart-raw' previously assumed that all non-binary MIME
parts could be successfully read by assuming that they were UTF-8
encoded. This was demonstrated to be wrong, specifically when a part was
marked as ISO8859-1 and included accented characters (which were
incorrectly rendered as a result).
Rather than assuming UTF-8, attempt to use the part's declared charset
when reading it, falling back to US-ASCII if the declared charset is
unknown, unsupported or invalid.
Tomi Ollila [Mon, 25 Apr 2016 08:44:25 +0000 (11:44 +0300)]
NEWS: mention try-emacs-mua
A non-technical introduction for users who read NEWS to have better
chance to find ./devel/notmuch-emacs-mua when they test or experiment
with notmuch emacs MUA next time.
To fully complete the ghost-on-removal-when-shared-thread-exists
proposal, we need to clear all ghost messages when the last active
message is removed from a thread.
Amended by db: Remove the last test of T530, as it no longer makes sense
if we are garbage collecting ghost messages.
implement ghost-on-removal, the solution to T590-thread-breakage.sh
that just adds a ghost message after removing each message.
It leaks information about whether we've ever seen a given message id,
but it's a fairly simple implementation.
Note that _resolve_message_id_to_thread_id already introduces new
message_ids to the database, so i think just searching for a given
message ID may introduce the same metadata leakage.
test thread breakage when messages are removed and re-added
This test (T590-thread-breakage.sh) has known-broken subtests.
If you have a two-message thread where message "B" is in-reply-to "A",
notmuch rightly sees this as a single thread.
But if you:
* remove "A" from the message store
* run "notmuch new"
* add "A" back into the message store
* re-run "notmuch new"
Then notmuch sees the messages as distinct threads.
This happens because if you insert "B" initially (before anything is
known about "A"), then a "ghost message" gets added to the database in
reference to "A" that is in the same thread, which "A" takes over when
it appears.
But if "A" is subsequently removed, no ghost message is retained, so
when "A" appears, it is treated as a new thread.
I see a few options to fix this:
ghost-on-removal
----------------
We could unilaterally add a ghost upon message removal. This has a
few disadvantages: the message index would leak information about what
messages the user has ever been exposed to, and we also create a
perpetually-growing dataset -- the ghosts can never be removed.
We could add a ghost upon message removal iff there are other
non-ghost messages with the same thread ID.
We'd also need to remove all ghost messages that share a thread when
the last non-ghost message in that thread is removed.
This still has a bit of information leakage, though: the message index
would reveal that i've seen a newer message in a thread, even if i had
deleted it from my message store
track-dependencies
------------------
rather than a simple "ghost-message" we could store all the (A,B)
message-reference pairs internally, showing which messages A reference
which other messages B.
Then removal of message X would require deleting all message-reference
pairs (X,B), and only deleting a ghost message if no (A,X) reference
pair exists.
This requires modifying the database by adding a new and fairly weird
table that would need to be indexed by both columns. I don't know
whether xapian has nice ways to do that.
scan-dependencies
-----------------
Without modifying the database, we could do something less efficient.
Upon removal of message X, we could scan the headers of all non-ghost
messages that share a thread with X. If any of those messages refers
to X, we would add a ghost message. If none of them do, then we would
just drop X entirely from the table.
---------------------
One risk of attempted fixes to this problem is that we could fail to
remove the search term indexes entirely. This test contains
additional subtests to guard against that.
This test also ensures that the right number of ghost messages exist
in each situation; this will help us ensure we don't accumulate ghosts
indefinitely or leak too much information about what messages we've
seen or not seen, while still making it easy to reassemble threads
when messages come in out-of-order.
Jani Nikula [Sun, 10 Apr 2016 19:43:22 +0000 (22:43 +0300)]
lib: fix handling of one character long directory names at top level
The code to skip multiple slashes in _notmuch_database_split_path()
skips back one character too much. This is compensated by a +1 in the
length parameter to the strndup() call. Mostly this works fine, but if
the path is to a file under a top level directory with one character
long name, the directory part is mistaken to be part of the file name
(slash == path in code). The returned directory name will be the empty
string and the basename will be the full path, breaking the indexing
logic in notmuch new.
Fix the multiple slash skipping to keep the slash variable pointing at
the last slash, and adjust strndup() accordingly.
David Bremner [Sat, 9 Apr 2016 01:49:50 +0000 (22:49 -0300)]
test: cope with glass backend file naming variations
In several places in the test suite we intentionally corrupt the Xapian
database in order to test error handling. This corruption is specific to
the on-disk organization of the database, and that changed with the
glass backend. We use the previously computed default backend to make
the tests adapt to changing names.
David Bremner [Sat, 9 Apr 2016 01:49:49 +0000 (22:49 -0300)]
configure: add test for default xapian backend
This is mainly for the test suite. We already expect the tests to be
run in the same environment as configure was run, at least to get the
name of the python interpreter. So we are not really imposing a new
restriction.
David Bremner [Sat, 9 Apr 2016 01:49:48 +0000 (22:49 -0300)]
test/atomicity: guard chert-only optimization
This should potentially be updated to have an equivalent optimization
for the glass backend, but it in my unscientific tests, the glass backend
without the optimization is faster then the chert backend with.
Please put my address in CC when replying. Thanks!
From 4b9ab261a0ea8a31065e310c5150f522be86d37b Mon Sep 17 00:00:00 2001
From: stefan <aeuii@posteo.de>
Date: Fri, 8 Apr 2016 22:47:06 +0200
Subject: [PATCH] emacs: make use of `message-make-from'
David Edmondson [Sun, 3 Apr 2016 15:38:22 +0000 (16:38 +0100)]
emacs: Always insert crypto buttons.
When no decryption or signature examination is
happening (i.e. `notmuch-crypto-process-mime' is `nil') insert buttons
that indicate this, rather than remaining silent.
David Edmondson [Sun, 3 Apr 2016 16:04:38 +0000 (17:04 +0100)]
emacs: Allow part preferences to depend on message content.
Currently the preference for which sub-part of a multipart/alternative
part is shown is global. Allow to the user to override the settings on a
per-message basis by providing the ability to call a function that has
access to the message to return the discouraged type list.