notmuch/configure, branch debian/unstable

lib: replace some uses of Query::MatchAll with a thread-safe alternative

2023-03-31T11:11:39Z

This replaces two instances of Xapian::Query::MatchAll with the equivalent but thread-safe alternative Xapian::Query(std::string()). Xapian::Query::MatchAll maintains an internal pointer to a refcounted Xapian::Internal::QueryTerm. None of this is thread-safe but that wouldn't be an issue if Xapian::Query::MatchAll wasn't static. Because it's static, the refcounting goes awry when Notmuch is called from multiple threads. This is actually documented by Xapian: https://github.com/xapian/xapian/blob/4715de3a9fcee741587439dc3cc1d2ff01ffeaf2/xapian-core/include/xapian/query.h#L65 While static, Xapian::Query::MatchNothing is safe because it doesn't maintain an internal object and as such, doesn't use references. Two best-effort tests making use of TSan were added to showcase the issue (I couldn't figure out a way to deterministically reproduce it without making an unmaintainable mess). First, when two databases are created in parallel, a query that uses Xapian::Query::MatchAll is made (lib/query.cc), resulting in the following backtrace on a segfault: #0 0x00007ffff76822af in Xapian::Query::get_terms_begin (this=0x7fffe80137f0) at api/query.cc:141 #1 0x00007ffff7f933f5 in _notmuch_query_cache_terms (query=0x7fffe80137c0) at lib/query.cc:176 #2 0x00007ffff7f93784 in _notmuch_query_ensure_parsed_xapian (query=0x7fffe80137c0) at lib/query.cc:225 #3 0x00007ffff7f9381a in _notmuch_query_ensure_parsed (query=0x7fffe80137c0) at lib/query.cc:260 #4 0x00007ffff7f93bfe in _notmuch_query_search_documents (query=0x7fffe80137c0, type=0x7ffff7fa9b1e "mail", out=0x7ffff666da18) at lib/query.cc:361 #5 0x00007ffff7f93ba4 in notmuch_query_search_messages (query=0x7fffe80137c0, out=0x7ffff666da18) at lib/query.cc:349 #6 0x00007ffff7f83d98 in notmuch_database_upgrade (notmuch=0x7fffe8000bd0, progress_notify=0x0, closure=0x0) at lib/database.cc:934 #7 0x00007ffff7fa110f in notmuch_database_create_with_config (database_path=0x7ffff666dcb0 "/tmp/notmuch.MZ2AGr", config_path=0x7ffff7faab3c "", profile=0x0, database=0x0, status_string=0x7ffff666dc90) at lib/open.cc:754 #8 0x00007ffff7fa0d6f in notmuch_database_create_verbose (path=0x7ffff666dcb0 "/tmp/notmuch.MZ2AGr", database=0x0, status_string=0x7ffff666dc90) at lib/open.cc:653 #9 0x00007ffff7fa0ceb in notmuch_database_create (path=0x7ffff666dcb0 "/tmp/notmuch.MZ2AGr", database=0x0) at lib/open.cc:637 ... Second, some queries would make use of Xapian::Query::MatchAll (lib/regexp-fields.cc), resulting in the following backtrace on a segfault: #0 0x00007f629828b690 in Xapian::Internal::QueryBranch::gather_terms (this=0x7f628800def0, void_terms=0x7f629726d5a0) at api/queryinternal.cc:1245 #1 0x00007f629828c260 in Xapian::Internal::QueryScaleWeight::gather_terms (this=0x7f628800df70, void_terms=0x7f629726d5a0) at api/queryinternal.cc:1434 #2 0x00007f629828b69f in Xapian::Internal::QueryBranch::gather_terms (this=0x7f628800dd90, void_terms=0x7f629726d5a0) at api/queryinternal.cc:1245 #3 0x00007f6298282571 in Xapian::Query::get_unique_terms_begin (this=0x7f628800dcd8) at api/query.cc:166 #4 0x00007f629841a59b in Xapian::Weight::Internal::accumulate_stats (this=0x7f628800dca0, subdb=..., rset=...) at weight/weightinternal.cc:86 #5 0x00007f62983c15ba in LocalSubMatch::prepare_match (this=0x7f628800df20, nowait=true, total_stats=...) at matcher/localsubmatch.cc:172 #6 0x00007f62983c8fcc in prepare_sub_matches (leaves=std::vector of length 1, capacity 1 = {...}, stats=...) at matcher/multimatch.cc:237 #7 0x00007f62983c98a3 in MultiMatch::MultiMatch (this=0x7f629726d9a0, db_=..., query_=..., qlen=3, omrset=0x0, collapse_max_=0, collapse_key_=4294967295, percent_cutoff_=0, weight_cutoff_=0, order_=Xapian::Enquire::ASCENDING, sort_key_=0, sort_by_=Xapian::Enquire::Internal::VAL, sort_value_forward_=true, time_limit_=0, stats=..., weight_=0x7f6288008d50, matchspies_=std::vector of length 0, capacity 0, have_sorter=false, have_mdecider=false) at matcher/multimatch.cc:353 #8 0x00007f629826fcba in Xapian::Enquire::Internal::get_mset (this=0x7f628800e0b0, first=0, maxitems=0, check_at_least=0, rset=0x0, mdecider=0x0) at api/omenquire.cc:569 #9 0x00007f629827181c in Xapian::Enquire::get_mset (this=0x7f629726db80, first=0, maxitems=0, check_at_least=0, rset=0x0, mdecider=0x0) at api/omenquire.cc:937 #10 0x00007f6298be529a in _notmuch_query_search_documents (query=0x7f6288009750, type=0x7f6298bfaafe "mail", out=0x7f629726dcc0) at lib/query.cc:447 #11 0x00007f6298be4ae8 in notmuch_query_search_messages (query=0x7f6288009750, out=0x7f629726dcc0) at lib/query.cc:349 ... Printing Xapian::Query::MatchAll->internal.px->_refs in these circumstances can help quickly identifying this scenario. This is motivated by some test frameworks (like Rust's Cargo) that runs unit tests in parallel and would easily encounter this issue, unless client code gates every call to Notmuch behind a lock. This is what can be expected from the tests when they fail: == stderr == +================== +WARNING: ThreadSanitizer: data race (pid=207931) + Read of size 1 at 0x7b10000001a0 by thread T2: + #0 memcpy (libtsan.so.2+0x62506) + #1 void std::__cxx11::basic_string, std::allocator >::_M_construct(char*, char*, std::forward_iterator_tag) [clone .isra.0] (libxapian.so.30+0x872b3) + + Previous write of size 8 at 0x7b10000001a0 by thread T1: + #0 operator new(unsigned long) (libtsan.so.2+0x8ba83) + #1 Xapian::Query::Query(std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned int) (libxapian.so.30+0x855cd) ...

configure: use pkg-config for gpgme

2023-01-10T12:29:23Z

GnuPG upstream has supported pkg-config since gpgme version 1.13 and gpg-error 1.33, and now prefers the use of pkg-config by default, instead of relying on gpg-error-config and gpgme-config. As of libgpg-error 1.46, upstream deliberately does not ship gpg-error-config by default. As of gpgme 1.18.0, upstream does not ship gpgme-config if gpg-error-config is also not present. Both of these versions of upstream libraries are in debian unstable now. To the extent that notmuch is dependent on GnuPG, it should follow GnuPG upstream's lead. Signed-off-by: Daniel Kahn Gillmor

test: replace aging OpenPGP key used in the test suite

2022-09-23T23:16:00Z

This replaces the old OpenPGPv4 key that is used in the test suite with a more modern OpenPGPv4 key. All cryptographic artifacts in the test suite are updated accordingly. Having old cryptographic artifacts in the test suite presents a problem once the old algorithms are rejected by contemporary implementations. For reference, this is the old key. sec rsa1024 2011-02-05 [SC] 5AEAB11F5E33DCE875DDB75B6D92612D94E46381 uid [ unknown] Notmuch Test Suite (INSECURE!) ssb rsa1024 2011-02-05 [E] And this is the new key. Note that is has the same shape, but uses Ed25519 and Cv25519 instead of 1024-bit RSA. sec ed25519 2022-09-07 [SC] 9A3AFE6C60065A148FD4B58A7E6ABE924645CC60 uid [ultimate] Notmuch Test Suite (INSECURE!) ssb cv25519 2022-09-07 [E]

test: define test_private_C

2022-06-25T15:55:02Z

When testing error handling, it is sometimes difficult to cover a particular error path deterministically. Introduce a test function to allow calling lower level functions directly.

configure: avoid warning with -Wall

2022-04-19T20:11:46Z

7228fe68 ("configure: restructure gmime cert validity checker code", 2022-04-09) restructured generated C code to repurpose it later on. This put usage of `validity` within an `#if`, resulting in an "unused warning" if that `#if` is not executed. Put the variable declariation inside the same if branch and, thus, quel the warning. Signed-off-by: Michael J Gruber

configure: clean up new test source and binary files.

2022-04-13T11:23:54Z

The previous source was renamed and a new binary generated in commit 8723e707c15f7b435f07f5d5ea693496bb9769bb.

test/smime: fix signature verification test with newer gmime.

2022-04-13T10:55:22Z

The extra machinery to check for the actual output format is justified by the possibility that distros may patch this newer output format into older versions of gmime. Amended-by: Michael J Gruber Signed-off-by: Michael J Gruber Amended-again-by: db

configure: restructure gmime cert validity checker code

2022-04-13T10:45:06Z

The goal is to generalize this to also check the output format of g_mime_certificate_get_email.

configure: check for ASAN support

2022-01-23T01:14:29Z

This will allow conditionally running tests that use the address sanitizer.

configure: Ignore more options that Fedora spec macros expect

2022-01-11T19:28:08Z

Signed-off-by: Michael J Gruber