David Bremner [Sat, 26 Aug 2023 10:45:36 +0000 (07:45 -0300)]
test: minimize impact of native compilation.
Native compilation is kindof useless in the test suite because we
throw away the cache after every subtest. The test suite could in
principle share an eln cache within a given test file; for now try to
minimize the amount of native-compilation. There is an intermittent
bug where emacs loses track of its default-directory; I suspect (but
have no proof) that bug is related to native compilation and/or race
conditions. This patch seems to prevent that bug (or at least reduce
its frequency).
David Bremner [Sun, 20 Aug 2023 17:32:02 +0000 (14:32 -0300)]
test/emacs: adapt to breaking change in Gnus defaults
As of Emacs 29.1, In-Reply-To is in the default value for
message-hidden-headers. We actually want to see that in the test
suite, so remove it again. To future proof the tests, fix a default
value for message-hidden-headers specifically for the test suite.
David Bremner [Thu, 20 Jul 2023 12:08:01 +0000 (09:08 -0300)]
lib/n_d_remove_message: do not remove unique filename
It is wasteful to remove a filename term when the whole message
document is about to be removed from the database. Profiling with perf
shows this takes a significant portion of the time when cleaning up
removed files in the database.
The logic of n_d_remove_message becomes a bit more convoluted here in
order to make the change minimal.
It is possible that this function can be further optimized, since the
expansion of filename terms into filenames is probably not needed
here.
David Bremner [Thu, 20 Jul 2023 12:08:00 +0000 (09:08 -0300)]
lib/message: check message type before deleting document
It isn't really clear how this worked before. Traversing the terms of
a document after deleting it from the database seems likely to be
undefined behaviour at best
David Bremner [Sun, 9 Apr 2023 14:26:26 +0000 (11:26 -0300)]
test: support testing notmuch as installed
We put some effort into testing the built copy rather than some
installed copy. On the other hand for people like packagers, testing
the installed copy is also of interest.
When NOTMUCH_TEST_INSTALLED is set to a nonempty value, tests do not
require a built notmuch tree or running configure.
Some of the tests marked as broken when running against installed
notmuch are probably fixable.
David Bremner [Sun, 9 Apr 2023 14:26:23 +0000 (11:26 -0300)]
test: treat undefined feature variables as 0
When running the test suite without building first, it is desirable to
have the tests consider these variables being undefined as equivalent
to the feature not being present, and in particular for the tests not
to generate errors.
David Bremner [Wed, 31 May 2023 11:19:18 +0000 (08:19 -0300)]
perf-test: update corpus signature
In the decade (!) since this corpus was last updated, the keyserver
network is essentially dead, and I have migrated gpg keys. Bump the
version number as a clean way of switching signatures. Also update the
instructions to suggest using "--locate-external-key" to download the
public key. By default this uses WKD, which is now supported for my
UID.
David Bremner [Sat, 27 May 2023 17:20:51 +0000 (14:20 -0300)]
test: add known broken test for message-id with embedded spaces.
According to my reading of RFC5322, there is an obsolete syntax for
Message-Id which permits folding whitespace (i.e. to be removed /
ignored by parsers). In [1] Paul Wise observed that notmuch removed
whitespace on indexing, but does not do any corresponding
normalization of queries. Mark the latter as a bug by adding a failing
test.
Michael J Gruber [Fri, 16 Jun 2023 11:19:26 +0000 (13:19 +0200)]
python: adjust legacy bindings to py 3.12
Py 3.12 finally pulled the plug on the `SafeConfigParser` class which
has been deprecated since py 3.2.
We use it in the legacy bindings only, so take the easy route of
importing `ConfigParser` as `SafeConfigParser` and monkey-patching so
that the class has the expected interface.
Paul Wise [Sun, 28 May 2023 02:29:46 +0000 (10:29 +0800)]
notmuch-mutt: check that the search cache Maildir is not a real Maildir
This prevents data loss when users configure the search cache Maildir to be a
real Maildir containing their real mail data, since the search cache Maildir
is expected to contain only symlinks to the real mail data.
David Bremner [Fri, 6 Jan 2023 00:02:05 +0000 (20:02 -0400)]
lib: parse index.as_text
We pre-parse into a list of compiled regular expressions to avoid
calling regexc on the hot (indexing) path. As explained in the code
comment, this cannot be done lazily with reasonable error reporting,
at least not without touching a lot of the code in index.cc.
Kevin Boulain [Thu, 2 Mar 2023 17:59:15 +0000 (18:59 +0100)]
lib: replace some uses of Query::MatchAll with a thread-safe alternative
This replaces two instances of Xapian::Query::MatchAll with the
equivalent but thread-safe alternative Xapian::Query(std::string()).
Xapian::Query::MatchAll maintains an internal pointer to a refcounted
Xapian::Internal::QueryTerm.
None of this is thread-safe but that wouldn't be an issue if
Xapian::Query::MatchAll wasn't static. Because it's static, the
refcounting goes awry when Notmuch is called from multiple threads.
This is actually documented by Xapian:
https://github.com/xapian/xapian/blob/4715de3a9fcee741587439dc3cc1d2ff01ffeaf2/xapian-core/include/xapian/query.h#L65
While static, Xapian::Query::MatchNothing is safe because it doesn't
maintain an internal object and as such, doesn't use references.
Two best-effort tests making use of TSan were added to showcase the
issue (I couldn't figure out a way to deterministically reproduce it
without making an unmaintainable mess).
First, when two databases are created in parallel, a query that uses
Xapian::Query::MatchAll is made (lib/query.cc), resulting in the
following backtrace on a segfault:
#0 0x00007ffff76822af in Xapian::Query::get_terms_begin (this=0x7fffe80137f0) at api/query.cc:141
#1 0x00007ffff7f933f5 in _notmuch_query_cache_terms (query=0x7fffe80137c0) at lib/query.cc:176
#2 0x00007ffff7f93784 in _notmuch_query_ensure_parsed_xapian (query=0x7fffe80137c0) at lib/query.cc:225
#3 0x00007ffff7f9381a in _notmuch_query_ensure_parsed (query=0x7fffe80137c0) at lib/query.cc:260
#4 0x00007ffff7f93bfe in _notmuch_query_search_documents (query=0x7fffe80137c0, type=0x7ffff7fa9b1e "mail", out=0x7ffff666da18) at lib/query.cc:361
#5 0x00007ffff7f93ba4 in notmuch_query_search_messages (query=0x7fffe80137c0, out=0x7ffff666da18) at lib/query.cc:349
#6 0x00007ffff7f83d98 in notmuch_database_upgrade (notmuch=0x7fffe8000bd0, progress_notify=0x0, closure=0x0) at lib/database.cc:934
#7 0x00007ffff7fa110f in notmuch_database_create_with_config (database_path=0x7ffff666dcb0 "/tmp/notmuch.MZ2AGr", config_path=0x7ffff7faab3c "", profile=0x0, database=0x0, status_string=0x7ffff666dc90) at lib/open.cc:754
#8 0x00007ffff7fa0d6f in notmuch_database_create_verbose (path=0x7ffff666dcb0 "/tmp/notmuch.MZ2AGr", database=0x0, status_string=0x7ffff666dc90) at lib/open.cc:653
#9 0x00007ffff7fa0ceb in notmuch_database_create (path=0x7ffff666dcb0 "/tmp/notmuch.MZ2AGr", database=0x0) at lib/open.cc:637
...
Second, some queries would make use of Xapian::Query::MatchAll
(lib/regexp-fields.cc), resulting in the following backtrace on a
segfault:
#0 0x00007f629828b690 in Xapian::Internal::QueryBranch::gather_terms (this=0x7f628800def0, void_terms=0x7f629726d5a0) at api/queryinternal.cc:1245
#1 0x00007f629828c260 in Xapian::Internal::QueryScaleWeight::gather_terms (this=0x7f628800df70, void_terms=0x7f629726d5a0) at api/queryinternal.cc:1434
#2 0x00007f629828b69f in Xapian::Internal::QueryBranch::gather_terms (this=0x7f628800dd90, void_terms=0x7f629726d5a0) at api/queryinternal.cc:1245
#3 0x00007f6298282571 in Xapian::Query::get_unique_terms_begin (this=0x7f628800dcd8) at api/query.cc:166
#4 0x00007f629841a59b in Xapian::Weight::Internal::accumulate_stats (this=0x7f628800dca0, subdb=..., rset=...) at weight/weightinternal.cc:86
#5 0x00007f62983c15ba in LocalSubMatch::prepare_match (this=0x7f628800df20, nowait=true, total_stats=...) at matcher/localsubmatch.cc:172
#6 0x00007f62983c8fcc in prepare_sub_matches (leaves=std::vector of length 1, capacity 1 = {...}, stats=...) at matcher/multimatch.cc:237
#7 0x00007f62983c98a3 in MultiMatch::MultiMatch (this=0x7f629726d9a0, db_=..., query_=..., qlen=3, omrset=0x0, collapse_max_=0, collapse_key_=4294967295, percent_cutoff_=0, weight_cutoff_=0, order_=Xapian::Enquire::ASCENDING, sort_key_=0, sort_by_=Xapian::Enquire::Internal::VAL, sort_value_forward_=true, time_limit_=0, stats=..., weight_=0x7f6288008d50, matchspies_=std::vector of length 0, capacity 0, have_sorter=false, have_mdecider=false) at matcher/multimatch.cc:353
#8 0x00007f629826fcba in Xapian::Enquire::Internal::get_mset (this=0x7f628800e0b0, first=0, maxitems=0, check_at_least=0, rset=0x0, mdecider=0x0) at api/omenquire.cc:569
#9 0x00007f629827181c in Xapian::Enquire::get_mset (this=0x7f629726db80, first=0, maxitems=0, check_at_least=0, rset=0x0, mdecider=0x0) at api/omenquire.cc:937
#10 0x00007f6298be529a in _notmuch_query_search_documents (query=0x7f6288009750, type=0x7f6298bfaafe "mail", out=0x7f629726dcc0) at lib/query.cc:447
#11 0x00007f6298be4ae8 in notmuch_query_search_messages (query=0x7f6288009750, out=0x7f629726dcc0) at lib/query.cc:349
...
Printing Xapian::Query::MatchAll->internal.px->_refs in these
circumstances can help quickly identifying this scenario.
This is motivated by some test frameworks (like Rust's Cargo) that
runs unit tests in parallel and would easily encounter this issue,
unless client code gates every call to Notmuch behind a lock.
This is what can be expected from the tests when they fail:
== stderr ==
+==================
+WARNING: ThreadSanitizer: data race (pid=207931)
+ Read of size 1 at 0x7b10000001a0 by thread T2:
+ #0 memcpy <null> (libtsan.so.2+0x62506)
+ #1 void std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::_M_construct<char*>(char*, char*, std::forward_iterator_tag) [clone .isra.0] <null> (libxapian.so.30+0x872b3)
+
+ Previous write of size 8 at 0x7b10000001a0 by thread T1:
+ #0 operator new(unsigned long) <null> (libtsan.so.2+0x8ba83)
+ #1 Xapian::Query::Query(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, unsigned int, unsigned int) <null> (libxapian.so.30+0x855cd)
...
Kevin Boulain [Wed, 29 Mar 2023 16:13:32 +0000 (18:13 +0200)]
lib/message-property: sync removed properties to the database
_notmuch_message_remove_all_properties wasn't syncing the message back
to the database but was still invalidating the metadata, giving the
impression the properties had actually been removed.
Also move the metadata invalidation to _notmuch_message_remove_terms
to be closer to what's done in _notmuch_message_modify_property and
_notmuch_message_remove_term.
Kevin Boulain [Wed, 29 Mar 2023 16:13:31 +0000 (18:13 +0200)]
test: reorganize tests and mark a few of them as broken
notmuch_message_remove_all_properties should have removed the
testkey1 = testvalue1
property but hasn't. Delay the execution of the corresponding test
to avoid updating a few tests that actually relied on the broken
behavior.
Kevin Boulain [Wed, 29 Mar 2023 16:19:57 +0000 (18:19 +0200)]
test: uncaught exception when editing properties of a removed message
These two functions don't fail gracefully when editing a removed
message:
BROKEN edit property on removed message without uncaught exception
--- T610-message-property.20.EXPECTED 2023-02-27 11:33:25.792764376 +0000
+++ T610-message-property.20.OUTPUT 2023-02-27 11:33:25.793764381 +0000
@@ -1,2 +1,3 @@
== stdout ==
== stderr ==
+terminate called after throwing an instance of 'Xapian::DocNotFoundError'
David Bremner [Tue, 27 Dec 2022 17:08:47 +0000 (13:08 -0400)]
lib: add better diagnostics for over long filenames.
Previously we just crashed with an internal error. With this change,
the caller can handle it better. Update notmuch-new so that it doesn't
crash with "unknown error code" because of this change.
jao [Tue, 13 Dec 2022 02:15:42 +0000 (02:15 +0000)]
emacs: notmuch-tree-outline-mode
With this mode, one can fold trees in the notmuch-tree buffer as if
they were outlines, using all the commands provided by
outline-minor-mode. We also define a couple of movement commands
that, optional, will ensure that only the thread around point is
unfolded.
The implementation is based on registering a :level property in the
messages p-list, that is then used by outline-minor-mode to to
recognise headers.
Amended by db: Copy docstring to manual and edit for presentation. Add
two tests. Fix typo "wether".
GnuPG upstream has supported pkg-config since gpgme version 1.13 and
gpg-error 1.33, and now prefers the use of pkg-config by default,
instead of relying on gpg-error-config and gpgme-config.
As of libgpg-error 1.46, upstream deliberately does not ship
gpg-error-config by default. As of gpgme 1.18.0, upstream does not
ship gpgme-config if gpg-error-config is also not present.
Both of these versions of upstream libraries are in debian unstable
now. To the extent that notmuch is dependent on GnuPG, it should
follow GnuPG upstream's lead.
Signed-off-by: Daniel Kahn Gillmor <dkg@fifthhorseman.net>
David Bremner [Sun, 4 Dec 2022 02:28:57 +0000 (22:28 -0400)]
lib/message: move xapian call inside try/catch block in _n_m_delete
The call to delete_document can throw exceptions (and can happen in
practice [1]), so catch the exception and extract the error
message. As a side effect, also move the call to _n_m_has_term inside
the try/catch. This should not change anything as that function
already traps any Xapian exceptions.
David Bremner [Sun, 16 Oct 2022 12:49:00 +0000 (09:49 -0300)]
CLI/git: use --exclude=false when calling notmuch-search
We use notmuch search in two places in notmuch-git.py: to find which
tags have a given prefix, and to see if message with given id exists
locally. In both cases we do not want the presence of exclude tags
(e.g. deleted) to change the results.
Michael J Gruber [Wed, 30 Nov 2022 13:58:34 +0000 (14:58 +0100)]
doc: mark `--output=summary` as default
`notmuch search` behaves differently depending on the output option: It
either outputs information pertaining to all threads with matching
messages (summary, threads) or to all matching messages (messages,
files, tags). The man page refres solely to the former in the main
description.
Help the user by clearly marking `summary` as the default output option.
David Bremner [Fri, 11 Nov 2022 21:48:31 +0000 (16:48 -0500)]
emacs/show: use n-s-i-headerline to update tags
Although this has more steps than the previous regular expression
search and replace, it should be more robust against changes in the
headerline format, such as the inclusion of duplicate numbers (which
broke the previous version).
Russell Sim [Tue, 20 Sep 2022 20:18:03 +0000 (22:18 +0200)]
emacs: move From header addition to after header intern
OTHER-HEADERS are expected to be passed as strings, to match the
implementation of `compose-mail'. But the "From" header is currently
expected to be passed as a symbol. Instead the "From" header can be
safely added after converting all the headers to symbols.
Robin Jarry [Tue, 18 Oct 2022 19:41:58 +0000 (21:41 +0200)]
cli: add options --offset and --limit to notmuch show
notmuch search does not output header values. However, when browsing
through a large email corpus, it can be time saving to be able to
paginate without running notmuch show for each message/thread.
Add --offset and --limit options to notmuch show. This is inspired from
commit 796b629c3b82 ("cli: add options --offset and --limit to notmuch
search").
Update man page, shell completion and add a test case to ensure it works
as expected.
Cc: Tim Culverhouse <tim@timculverhouse.com> Cc: Tomi Ollila <tomi.ollila@iki.fi> Signed-off-by: Robin Jarry <robin@jarry.cc>
Matt Armstrong [Thu, 13 Oct 2022 03:20:38 +0000 (20:20 -0700)]
emacs: fix dangling overlays in notmuch-search
notmuch-search-insert-authors now sets the evaporate property on the
ellipsis overlays. Emacs will delete them when the buffer contents
are zeroed out, which happens with `notmuch-refresh-buffer`. This
prevents them from being collapsed to zero-width overlays in position
1. See Emacs bug#58479. An upcoming change in Emacs will make these
dangling overlays visible to the user.
Tomi Ollila [Sun, 18 Sep 2022 13:05:50 +0000 (16:05 +0300)]
emacs: add notmuch-search-edit-search and notmuch-tree-edit-search
...and bind these to "E" in their respective keymaps.
Expected to be called interactively, then using read-from-minibuffer
with current search string as initial contents for editing.
(Noninteractive use makes little sense, but is supported.)
With this one can expand (as an opposite to limit) their
query and have e.g. (some of their) saved searches as search
"templates".
While at it, removed `(defvar notmuch-search-query-string)` from
notmuch-tree.el; it is unused (`notmuch-tree-basic-query` is used
instead).
Thanks to Jose Antonio Ortega Ruiz for his example for notmuch-tree
code, and better interactive use.
test: replace aging OpenPGP key used in the test suite
This replaces the old OpenPGPv4 key that is used in the test suite
with a more modern OpenPGPv4 key. All cryptographic artifacts in the
test suite are updated accordingly.
Having old cryptographic artifacts in the test suite presents a
problem once the old algorithms are rejected by contemporary
implementations.
For reference, this is the old key.
sec rsa1024 2011-02-05 [SC] 5AEAB11F5E33DCE875DDB75B6D92612D94E46381
uid [ unknown] Notmuch Test Suite <test_suite@notmuchmail.org> (INSECURE!)
ssb rsa1024 2011-02-05 [E]
And this is the new key. Note that is has the same shape, but uses Ed25519 and Cv25519 instead of 1024-bit RSA.
David Bremner [Sat, 20 Aug 2022 18:50:06 +0000 (11:50 -0700)]
test: add known broken test for indexing text/* attachments
The general problem of indexing attachments requires some help to turn
things into text, but (most?) text/* should be doable internally,
possibly with optimizations as for the text/html case.
David Bremner [Sat, 20 Aug 2022 18:50:05 +0000 (11:50 -0700)]
test: rename indexing corpus
The corpus is not really suitable for general indexing test since the
sole message is ignored (and will most likely continue to be ignored)
by notmuch-new.
David Bremner [Sun, 14 Aug 2022 15:02:59 +0000 (12:02 -0300)]
lib: add field processor for lastmod: prefix
By sharing the existing logic used by the sexp query parser, this
allows negative lastmod revisions to be interpreted as relative to the
most recent revision.
David Bremner [Sun, 14 Aug 2022 15:02:58 +0000 (12:02 -0300)]
lib: factor out lastmod range handling from sexp parser.
This will permit the re-use of the same logic in the infix query
parser. The location of the shared code in the infix side is for
consistency with the other shared parsing logic. It will make more
sense when a Xapian field processor is added for the lastmod prefix.
David Bremner [Fri, 29 Jul 2022 11:31:37 +0000 (08:31 -0300)]
lib/open: create database path in some cases
There is some duplication of code here, but not all of the locations
valid to find a database make sense to create. Furthermore we nead two
passes, so the control flow in _choose_database_path would get a bit
convoluted.
David Bremner [Thu, 28 Jul 2022 10:41:02 +0000 (06:41 -0400)]
lib/open: refactor call to mkdir into function
This makes the error handling available for re-use. Using
g_mkdir_with_parents also handles the case of a pre-existing
directory. This introduces new functionality, namely creating the
parent directories, which will be useful for creating directories like
'.local/share/notmuch/default'.
David Bremner [Tue, 26 Jul 2022 01:04:21 +0000 (21:04 -0400)]
test/path-config: set database.mail_root but not database.path
The failing "create database" test replicates a bug reported by Sean
Whitton [1]. The other two failures also look related to the database
being (re)created in the wrong place.
David Bremner [Tue, 26 Jul 2022 00:37:41 +0000 (20:37 -0400)]
test: add test for creating database in various configurations.
The existing database creation (via add_email_corpus) was always done
in the traditional configuration. The use of xapian-metadata is just
to portably ensure that there is a database created where we expect
there to be.
emacs/show: use read-shell-command instead of read-string
This enables auto-completion of commands, something which plain
read-string does not do. It's otherwise a drop-in
replacement. According to `C-h f`, read-shell-command was introduced
in Emacs 23.1 or earlier.
jao [Sun, 21 Aug 2022 03:06:18 +0000 (04:06 +0100)]
emacs: new notmuch-tree-process-exit-functions
Hook run when the tree insertion process finishes its job.
--
This patch supersedes <id:20220816214023.1523322-1-jao@gnu.org>, but
changing the new variable name.
Right now, it can be used for silly things like removing or changing
the the "End of search." hardcoded message in the tree buffer. But
also for more sophisticated things like folding all threads in add-ons
like my outline mode for tree buffers (to be submitted).
Michael J Gruber [Sun, 14 Aug 2022 15:22:17 +0000 (17:22 +0200)]
test: increase cffi timeout
By default, the test suite uses 2min for other tests and 5s for cffi
tests. Sporadically, this leads to test failures caused by the timeout
on slower or loaded test infrastructure (as seen on ppc64le in Fedora's
infrastructure during branch time).