]> git.notmuchmail.org Git - notmuch/log
7 hours agoMerge tag 'debian/0.38.1_pre0-1' master
David Bremner [Sun, 1 Oct 2023 11:26:17 +0000 (08:26 -0300)]
Merge tag 'debian/0.38.1_pre0-1'

notmuch release 0.38.1~pre0-1 for experimental (rc-buggy) [dgit]

[dgit distro=debian no-split --quilt=linear]

8 hours agodebian: finalize changelog for 0.38.1~pre0-1 release archive/debian/0.38.1_pre0-1 debian/0.38.1_pre0-1
David Bremner [Sun, 1 Oct 2023 11:16:48 +0000 (08:16 -0300)]
debian: finalize changelog for 0.38.1~pre0-1

Bonus typo fix.

8 hours agoMerge tag '0.38.1_pre0'
David Bremner [Sun, 1 Oct 2023 11:12:36 +0000 (08:12 -0300)]
Merge tag '0.38.1_pre0'

notmuch 0.38.1~pre0 release

8 hours agoemacs/show: special case toggling display of images 0.38.1_pre0
David Bremner [Sun, 24 Sep 2023 09:51:52 +0000 (06:51 -0300)]
emacs/show: special case toggling display of images

According to emacs upstream [1], we can't expect overlay invisibility
and images to get along. This commit uses the previously stashed
undisplayer functions to actually remove the images from the buffer.
When the image is toggled, it is essentially redisplayed from scratch,
using the previously stashed redisplay data.

[1]: https://lists.gnu.org/archive/html/emacs-devel/2023-08/msg00593.html

8 hours agoemacs: save image redisplay data during non-lazy display
David Bremner [Sun, 24 Sep 2023 09:51:51 +0000 (06:51 -0300)]
emacs: save image redisplay data during non-lazy display

This data will eventually be used to redisplay hidden images. A
certain amount of refactoring is done here to avoid code

8 hours agoemacs/show: save redisplay redisplay data when showing lazy part.
David Bremner [Sun, 24 Sep 2023 09:51:50 +0000 (06:51 -0300)]
emacs/show: save redisplay redisplay data when showing lazy part.

This data will be used to redisplay an image that is hidden by
deleting it from the buffer.  We cannot easily delay until the image
is hidden, as we won't have the original data at that point.

8 hours agoemacs: save undisplayer function for MIME parts
David Bremner [Sun, 24 Sep 2023 09:51:49 +0000 (06:51 -0300)]
emacs: save undisplayer function for MIME parts

For some kinds of MIME parts (at least images), our trickery with
overlays will not work, so save the more drastic function created by
Gnus that actually deletes the part from the buffer. In an ideal world
we would return this function as (part of) a value, but here the call
stack is too complicated for anything that simple, so we stash it in
the part plist and rely on that being preserved (unlike the mm handle,
which is transient).

8 days agoMerge branch 'release'
David Bremner [Sat, 23 Sep 2023 11:43:44 +0000 (08:43 -0300)]
Merge branch 'release'

8 days agoPass error message from GLib ini parser to CLI
David Bremner [Fri, 15 Sep 2023 12:50:04 +0000 (09:50 -0300)]
Pass error message from GLib ini parser to CLI

The function _notmuch_config_load_from_file is only called in two
places in open.cc. Update internal API to match the idiom in open.cc.
Adding a newline is needed for consistency with other status strings.

Based in part on a patch [1] from Eric Blake.

[1]: id:20230906153402.101471-1-eblake@redhat.com

8 days agotest: add known broken subtest for the bad config error message
David Bremner [Fri, 15 Sep 2023 12:50:03 +0000 (09:50 -0300)]
test: add known broken subtest for the bad config error message

This is a bit fragile w.r.t. glib changing their error message, but it
already helped me find one formatting bug, so for now I think it's
worth it, instead of just grepping for "UTF-8".

8 days agoCLI: exit with error when load_config returns an error.
David Bremner [Fri, 15 Sep 2023 12:50:02 +0000 (09:50 -0300)]
CLI: exit with error when load_config returns an error.

For now print a generic error message and exit with error on any
non-success code. Previously the code exited, but with exit code zero,
leading users / scripts to think the command had succeeded.

8 days agotest: add known broken test for bad utf8 in config
David Bremner [Fri, 15 Sep 2023 12:50:01 +0000 (09:50 -0300)]
test: add known broken test for bad utf8 in config

We should ideally print an informative error message, but at the very
least we should not exit with success.

8 days agodebian: start changelog for 0.38.1~pre0-1
David Bremner [Sat, 23 Sep 2023 11:16:20 +0000 (08:16 -0300)]
debian: start changelog for 0.38.1~pre0-1

8 days agoNEWS: add header for 0.38.1
David Bremner [Sat, 23 Sep 2023 11:14:21 +0000 (08:14 -0300)]
NEWS: add header for 0.38.1

Actual news items to be filled in after they are applied.

8 days agoversion: update to 0.38.1~pre0
David Bremner [Sat, 23 Sep 2023 11:12:06 +0000 (08:12 -0300)]
version: update to 0.38.1~pre0

Start preparations for point release

12 days ago.gitignore: ignore __pycache__
Eric Blake [Wed, 6 Sep 2023 15:26:08 +0000 (10:26 -0500)]
.gitignore: ignore __pycache__

Python likes to leave behind cache files; noticeable when doing an
in-tree build.

13 days agodevel/nmweb: read mail files in binary mode.
David Bremner [Sat, 16 Sep 2023 14:15:46 +0000 (11:15 -0300)]
devel/nmweb: read mail files in binary mode.

"ju" reported on IRC that browsing


crashed. The underlying issue is that python3 defaults to utf8
decoding files unless they are opened in binary mode. The file in
question (in the nmbug archive; it depends a bit on the routing the
message took) has

        Content-Type: text/plain; charset=iso-8859-1
        Content-Transfer-Encoding: 8bit

and some of it is not valid utf8.

2 weeks agoMerge branch 'release'
David Bremner [Wed, 13 Sep 2023 23:28:42 +0000 (20:28 -0300)]
Merge branch 'release'

2 weeks agodebian: upload 0.38-2: disable most autopkgtests archive/debian/0.38-2 debian/0.38-2
David Bremner [Wed, 13 Sep 2023 22:58:39 +0000 (19:58 -0300)]
debian: upload 0.38-2: disable most autopkgtests

2 weeks agoMerge branch 'release'
David Bremner [Tue, 12 Sep 2023 11:56:22 +0000 (08:56 -0300)]
Merge branch 'release'

2 weeks agobindings/python-cffi: clean up notmuch2.egg-info 0.38 archive/debian/0.38-1 debian/0.38-1
David Bremner [Sun, 10 Sep 2023 23:13:13 +0000 (20:13 -0300)]
bindings/python-cffi: clean up notmuch2.egg-info

This is created (at least) by the Debian build, but there seems no
harm in cleaning it for everyone.

2 weeks agodebian: note introduction of autopkgtests
David Bremner [Tue, 12 Sep 2023 11:33:35 +0000 (08:33 -0300)]
debian: note introduction of autopkgtests

Having autopkgtests changes Debian package migration (for better and
worse), so make a note when they were introduced to unstable.

2 weeks agoNEWS: quote function name
David Bremner [Sun, 10 Sep 2023 23:13:13 +0000 (20:13 -0300)]
NEWS: quote function name

This avoids spurious underline/italics when exporting to markdown.

2 weeks agoNEWS: set release date for 0.38
David Bremner [Sun, 10 Sep 2023 23:13:13 +0000 (20:13 -0300)]
NEWS: set release date for 0.38

2 weeks agobuild: clean up __pycache__ directories
David Bremner [Sun, 10 Sep 2023 23:13:13 +0000 (20:13 -0300)]
build: clean up __pycache__ directories

These are generated indirectly by certain uses of python in the build.

2 weeks agodebian: rely on main notmuch (dist)clean.
David Bremner [Sun, 10 Sep 2023 23:13:13 +0000 (20:13 -0300)]
debian: rely on main notmuch (dist)clean.

The python setuputils clean relys on including _notmuch_config.py,
which is cleaned up. Rather than relying on careful ordering, just do
all the cleaning from the GNU Make based build system.

2 weeks agobindings/python-cffi: clean up _notmuch_config.py
David Bremner [Sun, 10 Sep 2023 23:13:13 +0000 (20:13 -0300)]
bindings/python-cffi: clean up _notmuch_config.py

_notmuch_config.py is generated by configure, and cannot be cleaned up
by the current python build system, since it is imported as a module
by that same build system.

Use DISTCLEAN rather than CLEAN for consistency with other configure
related things.

2 weeks agodebian: changelog for 0.38-1
David Bremner [Mon, 11 Sep 2023 23:19:25 +0000 (20:19 -0300)]
debian: changelog for 0.38-1

2 weeks agoversion: bump to 0.38
David Bremner [Mon, 11 Sep 2023 22:56:21 +0000 (19:56 -0300)]
version: bump to 0.38

3 weeks agoNEWS: NEWS for 0.38
David Bremner [Sat, 9 Sep 2023 13:59:20 +0000 (10:59 -0300)]
NEWS: NEWS for 0.38

4 weeks agoMerge branch 'release'
David Bremner [Sun, 3 Sep 2023 13:20:40 +0000 (10:20 -0300)]
Merge branch 'release'

4 weeks agodebian: changelog for 0.38~rc2-1 0.38_rc2 archive/debian/0.38_rc2-1 debian/0.38_rc2-1
David Bremner [Sun, 3 Sep 2023 12:11:53 +0000 (09:11 -0300)]
debian: changelog for 0.38~rc2-1

4 weeks agoversion: bump to 0.31~rc2
David Bremner [Sun, 3 Sep 2023 12:09:22 +0000 (09:09 -0300)]
version: bump to 0.31~rc2

4 weeks agotest: suppress all interceptors in glib
Kevin Boulain [Wed, 30 Aug 2023 12:52:55 +0000 (14:52 +0200)]
test: suppress all interceptors in glib

On ppc64el, races are detected by TSan:
  WARNING: ThreadSanitizer: data race (pid=4520)
    Read of size 8 at 0x7ffff20016c0 by thread T1:
      #0 strlen ../../../../src/libsanitizer/sanitizer_common/sanitizer_common_interceptors.inc:386 (libtsan.so.2+0x77c0c)
      #1 strlen ../../../../src/libsanitizer/sanitizer_common/sanitizer_common_interceptors.inc:378 (libtsan.so.2+0x77c0c)
      #2 g_strdup ../../../glib/gstrfuncs.c:362 (libglib-2.0.so.0+0xa4ac4)

    Previous write of size 8 at 0x7ffff20016c0 by thread T2:
      #0 malloc ../../../../src/libsanitizer/tsan/tsan_interceptors_posix.cpp:647 (libtsan.so.2+0x471f0)
      #1 g_malloc ../../../glib/gmem.c:130 (libglib-2.0.so.0+0x7bb68)

    Location is heap block of size 20 at 0x7ffff20016c0 allocated by thread T2:
      #0 malloc ../../../../src/libsanitizer/tsan/tsan_interceptors_posix.cpp:647 (libtsan.so.2+0x471f0)
      #1 g_malloc ../../../glib/gmem.c:130 (libglib-2.0.so.0+0x7bb68)

This appears to be a false positive in GLib, as explained at
In short, a call to fstat fails under TSan and GLib's g_sterror will
intern the error message, which will be reused by other threads.

Since upstream appears to be aware that GLib doesn't play nicely with
TSan, suppress everything coming from the library instead of
maintaining a fine grained list.

Reported at

4 weeks agocompat: probe for strcasestr more thoroughly
Michael J Gruber [Sat, 26 Aug 2023 14:53:12 +0000 (16:53 +0200)]
compat: probe for strcasestr more thoroughly

Depending on compiler (gcc, g++, clang) and standard options (c99, c11),
string.h may or may not include strings.h, leading to possibly missing
or conflicting declarations of strcasestr.

Include both so that both detection and compilation phases use the same
(possibly optimised) implementations.

Suggested-by: Thomas Schneider <qsx@chaotikum.eu>
Suggested-by: Florian Weimer <fweimer@redhat.com>
Suggested-by: Tomi Ollila <tomi.ollila@iki.fi>
5 weeks agodebian: changelog for 0.38~rc1-1 0.38_rc1 archive/debian/0.38_rc1-1 debian/0.38_rc1-1
David Bremner [Sat, 26 Aug 2023 11:31:49 +0000 (08:31 -0300)]
debian: changelog for 0.38~rc1-1

5 weeks agoversion: bump to 0.38~rc1
David Bremner [Sat, 26 Aug 2023 11:29:45 +0000 (08:29 -0300)]
version: bump to 0.38~rc1

5 weeks agodebian: skip T810-tsan on ppc64el
David Bremner [Sat, 26 Aug 2023 11:28:33 +0000 (08:28 -0300)]
debian: skip T810-tsan on ppc64el

Hopefully just a temporary measure.

5 weeks agodebian: use architecture.mk
David Bremner [Sat, 26 Aug 2023 11:22:33 +0000 (08:22 -0300)]
debian: use architecture.mk

I plan to add further architecture dependent checks, which makes this
preferable to directly calling dpkg-architecture.

5 weeks agotest: minimize impact of native compilation.
David Bremner [Sat, 26 Aug 2023 10:45:36 +0000 (07:45 -0300)]
test: minimize impact of native compilation.

Native compilation is kindof useless in the test suite because we
throw away the cache after every subtest.  The test suite could in
principle share an eln cache within a given test file; for now try to
minimize the amount of native-compilation. There is an intermittent
bug where emacs loses track of its default-directory; I suspect (but
have no proof) that bug is related to native compilation and/or race
conditions. This patch seems to prevent that bug (or at least reduce
its frequency).

5 weeks agodebian: set suite to experimental archive/debian/0.38_rc0-1 debian/0.38_rc0-1
David Bremner [Thu, 24 Aug 2023 13:56:33 +0000 (10:56 -0300)]
debian: set suite to experimental

5 weeks agodoc: update copyright date 0.38_rc0
David Bremner [Thu, 24 Aug 2023 11:03:46 +0000 (08:03 -0300)]
doc: update copyright date

5 weeks agodebian: changelog for 0.38~rc0-1
David Bremner [Thu, 24 Aug 2023 11:00:29 +0000 (08:00 -0300)]
debian: changelog for 0.38~rc0-1

5 weeks agoNEWS: start NEWS for 0.38
David Bremner [Thu, 24 Aug 2023 10:58:51 +0000 (07:58 -0300)]
NEWS: start NEWS for 0.38

5 weeks agoversion: set up 0.38~rc0
David Bremner [Thu, 24 Aug 2023 10:58:19 +0000 (07:58 -0300)]
version: set up 0.38~rc0

5 weeks agotest/setup: ignore blank lines in generated config
David Bremner [Sun, 20 Aug 2023 15:23:19 +0000 (12:23 -0300)]
test/setup: ignore blank lines in generated config

The presense of the blank lines between sections depends on the
version of glib. Strip them before comparison.

5 weeks agoCLI/config: simulate top level comments when creating config
David Bremner [Sun, 20 Aug 2023 15:10:56 +0000 (12:10 -0300)]
CLI/config: simulate top level comments when creating config

According to discussion on


it looks like upstream will stop supporting top of file comments.

It is questionable whether we really need this feature, but for now
update notmuch-config to simulate it.

6 weeks agotest/emacs: adapt to breaking change in Gnus defaults
David Bremner [Sun, 20 Aug 2023 17:32:02 +0000 (14:32 -0300)]
test/emacs: adapt to breaking change in Gnus defaults

As of Emacs 29.1, In-Reply-To is in the default value for
message-hidden-headers. We actually want to see that in the test
suite, so remove it again. To future proof the tests, fix a default
value for message-hidden-headers specifically for the test suite.

2 months agolib/n_d_remove_message: do not remove unique filename
David Bremner [Thu, 20 Jul 2023 12:08:01 +0000 (09:08 -0300)]
lib/n_d_remove_message: do not remove unique filename

It is wasteful to remove a filename term when the whole message
document is about to be removed from the database. Profiling with perf
shows this takes a significant portion of the time when cleaning up
removed files in the database.

The logic of n_d_remove_message becomes a bit more convoluted here in
order to make the change minimal.

It is possible that this function can be further optimized, since the
expansion of filename terms into filenames is probably not needed

2 months agolib/message: check message type before deleting document
David Bremner [Thu, 20 Jul 2023 12:08:00 +0000 (09:08 -0300)]
lib/message: check message type before deleting document

It isn't really clear how this worked before. Traversing the terms of
a document after deleting it from the database seems likely to be
undefined behaviour at best

2 months agodebian: add autopkgtests
David Bremner [Sun, 9 Apr 2023 14:26:27 +0000 (11:26 -0300)]
debian: add autopkgtests

We generate output to stderr for BROKEN tests, which are not failures,
so tell the test runner not to fail because of output on stderr.

2 months agotest: support testing notmuch as installed
David Bremner [Sun, 9 Apr 2023 14:26:26 +0000 (11:26 -0300)]
test: support testing notmuch as installed

We put some effort into testing the built copy rather than some
installed copy. On the other hand for people like packagers, testing
the installed copy is also of interest.

When NOTMUCH_TEST_INSTALLED is set to a nonempty value, tests do not
require a built notmuch tree or running configure.

Some of the tests marked as broken when running against installed
notmuch are probably fixable.

2 months agotest: Guess a value for NOTMUCH_PYTHON
David Bremner [Sun, 9 Apr 2023 14:26:25 +0000 (11:26 -0300)]
test: Guess a value for NOTMUCH_PYTHON

python3 will work for many people, and reduce the friction to running
the tests without running configure first.

2 months agotest: check for empty/missing files in test_expect_equal_message_body
David Bremner [Sun, 9 Apr 2023 14:26:24 +0000 (11:26 -0300)]
test: check for empty/missing files in test_expect_equal_message_body

Messages can have empty bodies, but empty files are not messages.

2 months agotest: treat undefined feature variables as 0
David Bremner [Sun, 9 Apr 2023 14:26:23 +0000 (11:26 -0300)]
test: treat undefined feature variables as 0

When running the test suite without building first, it is desirable to
have the tests consider these variables being undefined as equivalent
to the feature not being present, and in particular for the tests not
to generate errors.

2 months agoschemata: document 'excluded' field in structured output
Tim Culverhouse [Sun, 7 May 2023 00:59:27 +0000 (19:59 -0500)]
schemata: document 'excluded' field in structured output

Include the 'excluded' field in the structured output schema.

Signed-off-by: Tim Culverhouse <tim@timculverhouse.com>
2 months agoperf-test: update corpus signature
David Bremner [Wed, 31 May 2023 11:19:18 +0000 (08:19 -0300)]
perf-test: update corpus signature

In the decade (!) since this corpus was last updated, the keyserver
network is essentially dead, and I have migrated gpg keys. Bump the
version number as a clean way of switching signatures. Also update the
instructions to suggest using "--locate-external-key" to download the
public key. By default this uses WKD, which is now supported for my

2 months agodoc/lib: clarify ownership for notmuch_database_get_revision
David Bremner [Mon, 29 May 2023 11:01:40 +0000 (08:01 -0300)]
doc/lib: clarify ownership for notmuch_database_get_revision

The ownership is implicit in the const declaration (I think!), but
that does not show up in the doxygen generated API docs.

2 months agotest: add known broken test for message-id with embedded spaces.
David Bremner [Sat, 27 May 2023 17:20:51 +0000 (14:20 -0300)]
test: add known broken test for message-id with embedded spaces.

According to my reading of RFC5322, there is an obsolete syntax for
Message-Id which permits folding whitespace (i.e. to be removed /
ignored by parsers). In [1] Paul Wise observed that notmuch removed
whitespace on indexing, but does not do any corresponding
normalization of queries. Mark the latter as a bug by adding a failing

[1]: id:20230409044143.4041560-1-pabs3@bonedaddy.net

2 months agopython: adjust legacy bindings to py 3.12
Michael J Gruber [Fri, 16 Jun 2023 11:19:26 +0000 (13:19 +0200)]
python: adjust legacy bindings to py 3.12

Py 3.12 finally pulled the plug on the `SafeConfigParser` class which
has been deprecated since py 3.2.

We use it in the legacy bindings only, so take the easy route of
importing `ConfigParser` as `SafeConfigParser` and monkey-patching so
that the class has the expected interface.

4 months agoruby: db: reorganize initializer
Felipe Contreras [Fri, 31 Mar 2023 20:40:50 +0000 (14:40 -0600)]
ruby: db: reorganize initializer

In order to make it more extensible.

No functional changes.

Signed-off-by: Felipe Contreras <felipe.contreras@gmail.com>
4 months agonotmuch-mutt: check that the search cache Maildir is not a real Maildir
Paul Wise [Sun, 28 May 2023 02:29:46 +0000 (10:29 +0800)]
notmuch-mutt: check that the search cache Maildir is not a real Maildir

This prevents data loss when users configure the search cache Maildir to be a
real Maildir containing their real mail data, since the search cache Maildir
is expected to contain only symlinks to the real mail data.

Prevents: <ZCsQBNmbzwkvbpHA@localhost.localdomain>

4 months agonotmuch-mutt: do not clear search cache Maildir when nothing is found
Paul Wise [Sun, 28 May 2023 02:29:45 +0000 (10:29 +0800)]
notmuch-mutt: do not clear search cache Maildir when nothing is found

The previous results might be useful to the user but
an empty directory definitely isn't useful.

4 months agonotmuch-mutt: clarify the empty Maildir function operates on search caches
Paul Wise [Sun, 28 May 2023 02:29:44 +0000 (10:29 +0800)]
notmuch-mutt: clarify the empty Maildir function operates on search caches

Rename the function and adjust the documentation comment.

4 months agonotmuch-mutt: replace extra command with notmuch thread search feature
Paul Wise [Sun, 9 Apr 2023 04:41:43 +0000 (12:41 +0800)]
notmuch-mutt: replace extra command with notmuch thread search feature

This should be be slightly faster since it avoids forking a shell
and is less code in and less dependencies for the script.

Since String::ShellQuote isn't used elsewhere, drop mention of it.

4 months agonotmuch-mutt: fix Xapian query construction
Paul Wise [Sun, 9 Apr 2023 04:41:42 +0000 (12:41 +0800)]
notmuch-mutt: fix Xapian query construction

Spaces need to be stripped when querying the Message-Id,
because notmuch stores them in Xapian with spaces stripped.

All double-quote characters need to be doubled to escape them,
otherwise they will be added as extra query terms outside the id.

4 months agonotmuch-mutt: convert ISO-8859-15 copyright statement to UTF-8
Paul Wise [Tue, 23 May 2023 22:30:21 +0000 (06:30 +0800)]
notmuch-mutt: convert ISO-8859-15 copyright statement to UTF-8

Suggested-by: isutf8 & check-all-the-things
4 months agoruby: query: fix get sort
Felipe Contreras [Mon, 3 Apr 2023 20:27:21 +0000 (14:27 -0600)]
ruby: query: fix get sort

The order was wrong, right now `query.sort` doesn't return a number.

Signed-off-by: Felipe Contreras <felipe.contreras@gmail.com>
4 months agovim: doc: minor spelling fix
John Gliksberg [Wed, 3 May 2023 22:12:20 +0000 (16:12 -0600)]
vim: doc: minor spelling fix

Signed-off-by: Felipe Contreras <felipe.contreras@gmail.com>
4 months agovim: doc: small fixes
Felipe Contreras [Wed, 3 May 2023 22:12:19 +0000 (16:12 -0600)]
vim: doc: small fixes

Signed-off-by: Felipe Contreras <felipe.contreras@gmail.com>
5 months agodoc: fix typos
Jakub Wilk [Thu, 13 Apr 2023 13:52:11 +0000 (15:52 +0200)]
doc: fix typos

5 months agoruby: remove FileNames object
Felipe Contreras [Mon, 27 Mar 2023 21:59:42 +0000 (15:59 -0600)]
ruby: remove FileNames object

Not used anymore now that we return an array of strings directly.

Signed-off-by: Felipe Contreras <felipe.contreras@gmail.com>
5 months agoruby: filenames: return string array directly
Felipe Contreras [Mon, 27 Mar 2023 21:59:41 +0000 (15:59 -0600)]
ruby: filenames: return string array directly

Signed-off-by: Felipe Contreras <felipe.contreras@gmail.com>
5 months agoruby: add filenames helper
Felipe Contreras [Mon, 27 Mar 2023 21:59:40 +0000 (15:59 -0600)]
ruby: add filenames helper

Right now it doesn't do much, but it will help for further

Signed-off-by: Felipe Contreras <felipe.contreras@gmail.com>
5 months agolib: index attachments with mime types matching index.as_text
David Bremner [Fri, 6 Jan 2023 00:02:06 +0000 (20:02 -0400)]
lib: index attachments with mime types matching index.as_text

Instead of skipping indexing all attachments, we check of a (user
configured) mime type that is indexable as text.

5 months agolib: parse index.as_text
David Bremner [Fri, 6 Jan 2023 00:02:05 +0000 (20:02 -0400)]
lib: parse index.as_text

We pre-parse into a list of compiled regular expressions to avoid
calling regexc on the hot (indexing) path.  As explained in the code
comment, this cannot be done lazily with reasonable error reporting,
at least not without touching a lot of the code in index.cc.

5 months agolib: add config key INDEX_AS_TEXT
David Bremner [Fri, 6 Jan 2023 00:02:04 +0000 (20:02 -0400)]
lib: add config key INDEX_AS_TEXT

Higher level processing as a list of regular expressions and
documentation will follow.

5 months agoruby: remove Tags object
Felipe Contreras [Wed, 22 Mar 2023 23:43:46 +0000 (17:43 -0600)]
ruby: remove Tags object

Not used anymore now that we return an array of strings directly.

Signed-off-by: Felipe Contreras <felipe.contreras@gmail.com>
5 months agoruby: tags: return string array directly
Felipe Contreras [Wed, 22 Mar 2023 23:43:45 +0000 (17:43 -0600)]
ruby: tags: return string array directly

Signed-off-by: Felipe Contreras <felipe.contreras@gmail.com>
5 months agoruby: add tags helper
Felipe Contreras [Wed, 22 Mar 2023 23:43:44 +0000 (17:43 -0600)]
ruby: add tags helper

Right now it doesn't do much, but it will help for further

Signed-off-by: Felipe Contreras <felipe.contreras@gmail.com>
6 months agolib: replace some uses of Query::MatchAll with a thread-safe alternative
Kevin Boulain [Thu, 2 Mar 2023 17:59:15 +0000 (18:59 +0100)]
lib: replace some uses of Query::MatchAll with a thread-safe alternative

This replaces two instances of Xapian::Query::MatchAll with the
equivalent but thread-safe alternative Xapian::Query(std::string()).
Xapian::Query::MatchAll maintains an internal pointer to a refcounted

None of this is thread-safe but that wouldn't be an issue if
Xapian::Query::MatchAll wasn't static. Because it's static, the
refcounting goes awry when Notmuch is called from multiple threads.
This is actually documented by Xapian:

While static, Xapian::Query::MatchNothing is safe because it doesn't
maintain an internal object and as such, doesn't use references.

Two best-effort tests making use of TSan were added to showcase the
issue (I couldn't figure out a way to deterministically reproduce it
without making an unmaintainable mess).

First, when two databases are created in parallel, a query that uses
Xapian::Query::MatchAll is made (lib/query.cc), resulting in the
following backtrace on a segfault:
  #0  0x00007ffff76822af in Xapian::Query::get_terms_begin (this=0x7fffe80137f0) at api/query.cc:141
  #1  0x00007ffff7f933f5 in _notmuch_query_cache_terms (query=0x7fffe80137c0) at lib/query.cc:176
  #2  0x00007ffff7f93784 in _notmuch_query_ensure_parsed_xapian (query=0x7fffe80137c0) at lib/query.cc:225
  #3  0x00007ffff7f9381a in _notmuch_query_ensure_parsed (query=0x7fffe80137c0) at lib/query.cc:260
  #4  0x00007ffff7f93bfe in _notmuch_query_search_documents (query=0x7fffe80137c0, type=0x7ffff7fa9b1e "mail", out=0x7ffff666da18) at lib/query.cc:361
  #5  0x00007ffff7f93ba4 in notmuch_query_search_messages (query=0x7fffe80137c0, out=0x7ffff666da18) at lib/query.cc:349
  #6  0x00007ffff7f83d98 in notmuch_database_upgrade (notmuch=0x7fffe8000bd0, progress_notify=0x0, closure=0x0) at lib/database.cc:934
  #7  0x00007ffff7fa110f in notmuch_database_create_with_config (database_path=0x7ffff666dcb0 "/tmp/notmuch.MZ2AGr", config_path=0x7ffff7faab3c "", profile=0x0, database=0x0, status_string=0x7ffff666dc90) at lib/open.cc:754
  #8  0x00007ffff7fa0d6f in notmuch_database_create_verbose (path=0x7ffff666dcb0 "/tmp/notmuch.MZ2AGr", database=0x0, status_string=0x7ffff666dc90) at lib/open.cc:653
  #9  0x00007ffff7fa0ceb in notmuch_database_create (path=0x7ffff666dcb0 "/tmp/notmuch.MZ2AGr", database=0x0) at lib/open.cc:637

Second, some queries would make use of Xapian::Query::MatchAll
(lib/regexp-fields.cc), resulting in the following backtrace on a
  #0  0x00007f629828b690 in Xapian::Internal::QueryBranch::gather_terms (this=0x7f628800def0, void_terms=0x7f629726d5a0) at api/queryinternal.cc:1245
  #1  0x00007f629828c260 in Xapian::Internal::QueryScaleWeight::gather_terms (this=0x7f628800df70, void_terms=0x7f629726d5a0) at api/queryinternal.cc:1434
  #2  0x00007f629828b69f in Xapian::Internal::QueryBranch::gather_terms (this=0x7f628800dd90, void_terms=0x7f629726d5a0) at api/queryinternal.cc:1245
  #3  0x00007f6298282571 in Xapian::Query::get_unique_terms_begin (this=0x7f628800dcd8) at api/query.cc:166
  #4  0x00007f629841a59b in Xapian::Weight::Internal::accumulate_stats (this=0x7f628800dca0, subdb=..., rset=...) at weight/weightinternal.cc:86
  #5  0x00007f62983c15ba in LocalSubMatch::prepare_match (this=0x7f628800df20, nowait=true, total_stats=...) at matcher/localsubmatch.cc:172
  #6  0x00007f62983c8fcc in prepare_sub_matches (leaves=std::vector of length 1, capacity 1 = {...}, stats=...) at matcher/multimatch.cc:237
  #7  0x00007f62983c98a3 in MultiMatch::MultiMatch (this=0x7f629726d9a0, db_=..., query_=..., qlen=3, omrset=0x0, collapse_max_=0, collapse_key_=4294967295, percent_cutoff_=0, weight_cutoff_=0, order_=Xapian::Enquire::ASCENDING, sort_key_=0, sort_by_=Xapian::Enquire::Internal::VAL, sort_value_forward_=true, time_limit_=0, stats=..., weight_=0x7f6288008d50, matchspies_=std::vector of length 0, capacity 0, have_sorter=false, have_mdecider=false) at matcher/multimatch.cc:353
  #8  0x00007f629826fcba in Xapian::Enquire::Internal::get_mset (this=0x7f628800e0b0, first=0, maxitems=0, check_at_least=0, rset=0x0, mdecider=0x0) at api/omenquire.cc:569
  #9  0x00007f629827181c in Xapian::Enquire::get_mset (this=0x7f629726db80, first=0, maxitems=0, check_at_least=0, rset=0x0, mdecider=0x0) at api/omenquire.cc:937
  #10 0x00007f6298be529a in _notmuch_query_search_documents (query=0x7f6288009750, type=0x7f6298bfaafe "mail", out=0x7f629726dcc0) at lib/query.cc:447
  #11 0x00007f6298be4ae8 in notmuch_query_search_messages (query=0x7f6288009750, out=0x7f629726dcc0) at lib/query.cc:349

Printing Xapian::Query::MatchAll->internal.px->_refs in these
circumstances can help quickly identifying this scenario.

This is motivated by some test frameworks (like Rust's Cargo) that
runs unit tests in parallel and would easily encounter this issue,
unless client code gates every call to Notmuch behind a lock.

This is what can be expected from the tests when they fail:
   == stderr ==
  +WARNING: ThreadSanitizer: data race (pid=207931)
  +  Read of size 1 at 0x7b10000001a0 by thread T2:
  +    #0 memcpy <null> (libtsan.so.2+0x62506)
  +    #1 void std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::_M_construct<char*>(char*, char*, std::forward_iterator_tag) [clone .isra.0] <null> (libxapian.so.30+0x872b3)
  +  Previous write of size 8 at 0x7b10000001a0 by thread T1:
  +    #0 operator new(unsigned long) <null> (libtsan.so.2+0x8ba83)
  +    #1 Xapian::Query::Query(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, unsigned int, unsigned int) <null> (libxapian.so.30+0x855cd)

6 months agoruby: database: make path arg optional
Felipe Contreras [Thu, 23 Mar 2023 02:05:23 +0000 (20:05 -0600)]
ruby: database: make path arg optional

It can be automatically loaded from the configuration now.

Signed-off-by: Felipe Contreras <felipe.contreras@gmail.com>
6 months agoruby: use database_open_with_config
Felipe Contreras [Thu, 23 Mar 2023 02:05:22 +0000 (20:05 -0600)]
ruby: use database_open_with_config

Fixes warning:

  warning: ‘notmuch_database_open’ is deprecated: function deprecated as of libnotmuch 5.4

Signed-off-by: Felipe Contreras <felipe.contreras@gmail.com>
6 months agotest: add test for notmuch_message_remove_all_properties_with_prefix
Kevin Boulain [Wed, 29 Mar 2023 16:13:33 +0000 (18:13 +0200)]
test: add test for notmuch_message_remove_all_properties_with_prefix

It wasn't covered, though it shares most of its implementation with

6 months agolib/message-property: sync removed properties to the database
Kevin Boulain [Wed, 29 Mar 2023 16:13:32 +0000 (18:13 +0200)]
lib/message-property: sync removed properties to the database

_notmuch_message_remove_all_properties wasn't syncing the message back
to the database but was still invalidating the metadata, giving the
impression the properties had actually been removed.

Also move the metadata invalidation to _notmuch_message_remove_terms
to be closer to what's done in _notmuch_message_modify_property and

6 months agotest: reveal notmuch_message_remove_all_properties as broken
David Bremner [Thu, 30 Mar 2023 10:56:17 +0000 (07:56 -0300)]
test: reveal notmuch_message_remove_all_properties as broken

Close and re-open the database to show that the removal is not
committed to the database.

6 months agotest: reorganize tests and mark a few of them as broken
Kevin Boulain [Wed, 29 Mar 2023 16:13:31 +0000 (18:13 +0200)]
test: reorganize tests and mark a few of them as broken

notmuch_message_remove_all_properties should have removed the
  testkey1 = testvalue1
property but hasn't. Delay the execution of the corresponding test
to avoid updating a few tests that actually relied on the broken

6 months agotest: remove unnecessary sorting
Kevin Boulain [Wed, 29 Mar 2023 16:13:30 +0000 (18:13 +0200)]
test: remove unnecessary sorting

The other tests rely on a stable output.

6 months agotest: display key name in property tests
Kevin Boulain [Wed, 29 Mar 2023 16:13:29 +0000 (18:13 +0200)]
test: display key name in property tests

To make the tests a bit easier to understand.

6 months agolib/message-property: catch xapian exceptions
Kevin Boulain [Wed, 29 Mar 2023 16:19:58 +0000 (18:19 +0200)]
lib/message-property: catch xapian exceptions

Since libnotmuch exposes a C interface there's no way for clients to
catch this.
Inspired by what's done for tags (see notmuch_message_remove_tag).

6 months agotest: uncaught exception when editing properties of a removed message
Kevin Boulain [Wed, 29 Mar 2023 16:19:57 +0000 (18:19 +0200)]
test: uncaught exception when editing properties of a removed message

These two functions don't fail gracefully when editing a removed
 BROKEN edit property on removed message without uncaught exception
        --- T610-message-property.20.EXPECTED   2023-02-27 11:33:25.792764376 +0000
        +++ T610-message-property.20.OUTPUT     2023-02-27 11:33:25.793764381 +0000
        @@ -1,2 +1,3 @@
         == stdout ==
         == stderr ==
        +terminate called after throwing an instance of 'Xapian::DocNotFoundError'

The other functions appear to be safe.

7 months agolib/notmuch: update example
Kevin Boulain [Mon, 27 Feb 2023 12:06:29 +0000 (13:06 +0100)]
lib/notmuch: update example

Likely missed in 86cbd215e, when notmuch_query_search_messages_st was
renamed to notmuch_query_search_messages.

7 months agoemacs/tree: use two argument form of setq-local
David Bremner [Tue, 21 Feb 2023 11:49:15 +0000 (07:49 -0400)]
emacs/tree: use two argument form of setq-local

Apparently the macro setq-local only takes two arguments in Emacs 26.1

7 months agolib: add better diagnostics for over long filenames.
David Bremner [Tue, 27 Dec 2022 17:08:47 +0000 (13:08 -0400)]
lib: add better diagnostics for over long filenames.

Previously we just crashed with an internal error. With this change,
the caller can handle it better. Update notmuch-new so that it doesn't
crash with "unknown error code" because of this change.

7 months agotest: add known broken test for diagnostics from over long filenames.
David Bremner [Tue, 27 Dec 2022 17:08:46 +0000 (13:08 -0400)]
test: add known broken test for diagnostics from over long filenames.

Previously we tested over long directory names, add similar testing
for over long filenames.

7 months agoemacs: notmuch-tree-outline-mode
jao [Tue, 13 Dec 2022 02:15:42 +0000 (02:15 +0000)]
emacs: notmuch-tree-outline-mode

With this mode, one can fold trees in the notmuch-tree buffer as if
they were outlines, using all the commands provided by
outline-minor-mode.  We also define a couple of movement commands
that, optional, will ensure that only the thread around point is

The implementation is based on registering a :level property in the
messages p-list, that is then used by outline-minor-mode to to
recognise headers.

Amended by db: Copy docstring to manual and edit for presentation. Add
two tests. Fix typo "wether".

8 months agoconfigure: use pkg-config for gpgme
Daniel Kahn Gillmor [Wed, 2 Nov 2022 16:15:12 +0000 (12:15 -0400)]
configure: use pkg-config for gpgme

GnuPG upstream has supported pkg-config since gpgme version 1.13 and
gpg-error 1.33, and now prefers the use of pkg-config by default,
instead of relying on gpg-error-config and gpgme-config.

As of libgpg-error 1.46, upstream deliberately does not ship
gpg-error-config by default.  As of gpgme 1.18.0, upstream does not
ship gpgme-config if gpg-error-config is also not present.

Both of these versions of upstream libraries are in debian unstable
now.  To the extent that notmuch is dependent on GnuPG, it should
follow GnuPG upstream's lead.

Signed-off-by: Daniel Kahn Gillmor <dkg@fifthhorseman.net>
8 months agotest: mark some tests as broken when run as root.
David Bremner [Wed, 4 Jan 2023 12:41:42 +0000 (08:41 -0400)]
test: mark some tests as broken when run as root.

File permission errors e.g., are hard to trigger as root.

9 months agolib/message: move xapian call inside try/catch block in _n_m_delete
David Bremner [Sun, 4 Dec 2022 02:28:57 +0000 (22:28 -0400)]
lib/message: move xapian call inside try/catch block in _n_m_delete

The call to delete_document can throw exceptions (and can happen in
practice [1]), so catch the exception and extract the error
message. As a side effect, also move the call to _n_m_has_term inside
the try/catch. This should not change anything as that function
already traps any Xapian exceptions.

[1]: id:wwuk039sk2p.fsf@chaotikum.eu

9 months agotest: add known broken test for exception handling in _n_m_delete
David Bremner [Sun, 4 Dec 2022 02:28:56 +0000 (22:28 -0400)]
test: add known broken test for exception handling in _n_m_delete

In [1], Thomas Schneider reported an uncaught Xapian exception when
running out of disk space. We generate the same exception via database

[1]: id:wwuk039sk2p.fsf@chaotikum.eu