| Age | Commit message (Collapse) | Author |
|---|
|
This reflects the addition of a new API functions,
notmuch_{messages,threads}_status
|
|
It is wrong most of the time in _notmuch_message_remove_terms, but
that function is too low level to know how to call
_n_m_invalidate_metadata with the right argument, at least not without
more extensive changes. This change merely makes the current behaviour
more obvious, since the other calls cannot have relied on metadata
being invalidated.
|
|
Regexes compiled with regcomp() need to be freed with regfree(). Do that
in a talloc destructor attached to the compiled regex array.
Amended by db: Use C style comment. Add blank line per uncrustify.
|
|
The overall goal is to allow clients to restart operations in
situations where that is the response recommended by the underlying
Xapian library.
Amended-by: db, added above explanation
|
|
Silences compiler warnings about redefining a macro.
|
|
While a number of errors can happen when iterating over threads
(DatabaseModifiedError, memory allocation errors, etc.), the API
currently cannot signal them to the caller, and either triggers an
internal error (aborting the caller) or returns NULL from
notmuch_threads_get() with no information on what actually went wrong.
Add a new public function notmuch_threads_status() - similar to
previously added notmuch_messages_status() - that allows propagating
those errors to the caller.
Use this to remove the INTERNAL_ERROR() in _notmuch_thread_create()
(triggered by T642).
Fixes: https://github.com/pazz/alot/issues/1460
Amended-By: db. Resolved whitespace disagreement between emacs and
uncrustify in favour of uncrustify.
|
|
The list destructor is always set when necessary (i.e. if the iterator
is successfully created), so there is never a need to invoke it
explicitly, even on failure.
|
|
If an open database is modified sufficiently by other callers, the open
instance becomes invalid and operations on it throw
DatabaseModifiedError. Per Xapian documentation, the caller is then
supposed to reopen the database and restart the query. This exception is
currently not handled in _notmuch_message_create(), leading to the
default handler abort()ing the process.
Catch this exception in _notmuch_message_create() and return an error
instead of crashing. Since the entire query becomes invalid - including
results that have already been read by the caller - this situation
cannot be handled by libnotmuch transparently. A new public function -
notmuch_messages_status() - is added to allow the callers to check
whether the messages iterator was exhausted or terminated early due to
a runtime error. This also allows memory allocation failure to be
signalled to the caller.
Amended-By: David Bremner <david@tethera.net>
[replace use of notmuch_messages_valid]
|
|
Allow distinguishing between commas separating authors and separating
first and last names.
Amended by db: reformat NEWS entry and commit message. Tweaked
whitespace in lib/thread.cc.
|
|
Follow-up of 6273966d, now that sfsexp 1.4.1 doesn't rely on globals
anymore by default (https://github.com/mjsottile/sfsexp/issues/21).
This simply defers the initial query generation to use the thread-safe
helper (xapian_query_match_all) instead of Xapian::Query::MatchAll.
|
|
Previously we both found a database and returned a message saying that
the database could not be found (along with a success code). This
change should prevent spurious error output.
|
|
qsort(3) does not promise stability, and recent versions of glibc have
been showing more unstable behaviour [2]. Michael Gruber observed [1] test
breakage due to changing output order for message properties.
We provide a sorting order of (key,value) pairs that _looks_ stable by
breaking ties based on value if keys are equal. Internally there may
be some instability in the case of duplicate (key,value) pairs, but it
should not be observable via the iterator API.
[1]: id:CAA19uiSHjVFmwH0pMC7WwDYCOSzu3yqNbuYhu3ZMeNNRh313eA@mail.gmail.com
[2]: id:87msv3i44u.fsf@oldenburg.str.redhat.com
|
|
The function _notmuch_config_load_from_file is only called in two
places in open.cc. Update internal API to match the idiom in open.cc.
Adding a newline is needed for consistency with other status strings.
Based in part on a patch [1] from Eric Blake.
[1]: id:20230906153402.101471-1-eblake@redhat.com
|
|
It is wasteful to remove a filename term when the whole message
document is about to be removed from the database. Profiling with perf
shows this takes a significant portion of the time when cleaning up
removed files in the database.
The logic of n_d_remove_message becomes a bit more convoluted here in
order to make the change minimal.
It is possible that this function can be further optimized, since the
expansion of filename terms into filenames is probably not needed
here.
|
|
It isn't really clear how this worked before. Traversing the terms of
a document after deleting it from the database seems likely to be
undefined behaviour at best
|
|
The ownership is implicit in the const declaration (I think!), but
that does not show up in the doxygen generated API docs.
|
|
Instead of skipping indexing all attachments, we check of a (user
configured) mime type that is indexable as text.
|
|
We pre-parse into a list of compiled regular expressions to avoid
calling regexc on the hot (indexing) path. As explained in the code
comment, this cannot be done lazily with reasonable error reporting,
at least not without touching a lot of the code in index.cc.
|
|
Higher level processing as a list of regular expressions and
documentation will follow.
|
|
This replaces two instances of Xapian::Query::MatchAll with the
equivalent but thread-safe alternative Xapian::Query(std::string()).
Xapian::Query::MatchAll maintains an internal pointer to a refcounted
Xapian::Internal::QueryTerm.
None of this is thread-safe but that wouldn't be an issue if
Xapian::Query::MatchAll wasn't static. Because it's static, the
refcounting goes awry when Notmuch is called from multiple threads.
This is actually documented by Xapian:
https://github.com/xapian/xapian/blob/4715de3a9fcee741587439dc3cc1d2ff01ffeaf2/xapian-core/include/xapian/query.h#L65
While static, Xapian::Query::MatchNothing is safe because it doesn't
maintain an internal object and as such, doesn't use references.
Two best-effort tests making use of TSan were added to showcase the
issue (I couldn't figure out a way to deterministically reproduce it
without making an unmaintainable mess).
First, when two databases are created in parallel, a query that uses
Xapian::Query::MatchAll is made (lib/query.cc), resulting in the
following backtrace on a segfault:
#0 0x00007ffff76822af in Xapian::Query::get_terms_begin (this=0x7fffe80137f0) at api/query.cc:141
#1 0x00007ffff7f933f5 in _notmuch_query_cache_terms (query=0x7fffe80137c0) at lib/query.cc:176
#2 0x00007ffff7f93784 in _notmuch_query_ensure_parsed_xapian (query=0x7fffe80137c0) at lib/query.cc:225
#3 0x00007ffff7f9381a in _notmuch_query_ensure_parsed (query=0x7fffe80137c0) at lib/query.cc:260
#4 0x00007ffff7f93bfe in _notmuch_query_search_documents (query=0x7fffe80137c0, type=0x7ffff7fa9b1e "mail", out=0x7ffff666da18) at lib/query.cc:361
#5 0x00007ffff7f93ba4 in notmuch_query_search_messages (query=0x7fffe80137c0, out=0x7ffff666da18) at lib/query.cc:349
#6 0x00007ffff7f83d98 in notmuch_database_upgrade (notmuch=0x7fffe8000bd0, progress_notify=0x0, closure=0x0) at lib/database.cc:934
#7 0x00007ffff7fa110f in notmuch_database_create_with_config (database_path=0x7ffff666dcb0 "/tmp/notmuch.MZ2AGr", config_path=0x7ffff7faab3c "", profile=0x0, database=0x0, status_string=0x7ffff666dc90) at lib/open.cc:754
#8 0x00007ffff7fa0d6f in notmuch_database_create_verbose (path=0x7ffff666dcb0 "/tmp/notmuch.MZ2AGr", database=0x0, status_string=0x7ffff666dc90) at lib/open.cc:653
#9 0x00007ffff7fa0ceb in notmuch_database_create (path=0x7ffff666dcb0 "/tmp/notmuch.MZ2AGr", database=0x0) at lib/open.cc:637
...
Second, some queries would make use of Xapian::Query::MatchAll
(lib/regexp-fields.cc), resulting in the following backtrace on a
segfault:
#0 0x00007f629828b690 in Xapian::Internal::QueryBranch::gather_terms (this=0x7f628800def0, void_terms=0x7f629726d5a0) at api/queryinternal.cc:1245
#1 0x00007f629828c260 in Xapian::Internal::QueryScaleWeight::gather_terms (this=0x7f628800df70, void_terms=0x7f629726d5a0) at api/queryinternal.cc:1434
#2 0x00007f629828b69f in Xapian::Internal::QueryBranch::gather_terms (this=0x7f628800dd90, void_terms=0x7f629726d5a0) at api/queryinternal.cc:1245
#3 0x00007f6298282571 in Xapian::Query::get_unique_terms_begin (this=0x7f628800dcd8) at api/query.cc:166
#4 0x00007f629841a59b in Xapian::Weight::Internal::accumulate_stats (this=0x7f628800dca0, subdb=..., rset=...) at weight/weightinternal.cc:86
#5 0x00007f62983c15ba in LocalSubMatch::prepare_match (this=0x7f628800df20, nowait=true, total_stats=...) at matcher/localsubmatch.cc:172
#6 0x00007f62983c8fcc in prepare_sub_matches (leaves=std::vector of length 1, capacity 1 = {...}, stats=...) at matcher/multimatch.cc:237
#7 0x00007f62983c98a3 in MultiMatch::MultiMatch (this=0x7f629726d9a0, db_=..., query_=..., qlen=3, omrset=0x0, collapse_max_=0, collapse_key_=4294967295, percent_cutoff_=0, weight_cutoff_=0, order_=Xapian::Enquire::ASCENDING, sort_key_=0, sort_by_=Xapian::Enquire::Internal::VAL, sort_value_forward_=true, time_limit_=0, stats=..., weight_=0x7f6288008d50, matchspies_=std::vector of length 0, capacity 0, have_sorter=false, have_mdecider=false) at matcher/multimatch.cc:353
#8 0x00007f629826fcba in Xapian::Enquire::Internal::get_mset (this=0x7f628800e0b0, first=0, maxitems=0, check_at_least=0, rset=0x0, mdecider=0x0) at api/omenquire.cc:569
#9 0x00007f629827181c in Xapian::Enquire::get_mset (this=0x7f629726db80, first=0, maxitems=0, check_at_least=0, rset=0x0, mdecider=0x0) at api/omenquire.cc:937
#10 0x00007f6298be529a in _notmuch_query_search_documents (query=0x7f6288009750, type=0x7f6298bfaafe "mail", out=0x7f629726dcc0) at lib/query.cc:447
#11 0x00007f6298be4ae8 in notmuch_query_search_messages (query=0x7f6288009750, out=0x7f629726dcc0) at lib/query.cc:349
...
Printing Xapian::Query::MatchAll->internal.px->_refs in these
circumstances can help quickly identifying this scenario.
This is motivated by some test frameworks (like Rust's Cargo) that
runs unit tests in parallel and would easily encounter this issue,
unless client code gates every call to Notmuch behind a lock.
This is what can be expected from the tests when they fail:
== stderr ==
+==================
+WARNING: ThreadSanitizer: data race (pid=207931)
+ Read of size 1 at 0x7b10000001a0 by thread T2:
+ #0 memcpy <null> (libtsan.so.2+0x62506)
+ #1 void std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::_M_construct<char*>(char*, char*, std::forward_iterator_tag) [clone .isra.0] <null> (libxapian.so.30+0x872b3)
+
+ Previous write of size 8 at 0x7b10000001a0 by thread T1:
+ #0 operator new(unsigned long) <null> (libtsan.so.2+0x8ba83)
+ #1 Xapian::Query::Query(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, unsigned int, unsigned int) <null> (libxapian.so.30+0x855cd)
...
|
|
_notmuch_message_remove_all_properties wasn't syncing the message back
to the database but was still invalidating the metadata, giving the
impression the properties had actually been removed.
Also move the metadata invalidation to _notmuch_message_remove_terms
to be closer to what's done in _notmuch_message_modify_property and
_notmuch_message_remove_term.
|
|
Since libnotmuch exposes a C interface there's no way for clients to
catch this.
Inspired by what's done for tags (see notmuch_message_remove_tag).
|
|
Likely missed in 86cbd215e, when notmuch_query_search_messages_st was
renamed to notmuch_query_search_messages.
|
|
Previously we just crashed with an internal error. With this change,
the caller can handle it better. Update notmuch-new so that it doesn't
crash with "unknown error code" because of this change.
|
|
The call to delete_document can throw exceptions (and can happen in
practice [1]), so catch the exception and extract the error
message. As a side effect, also move the call to _n_m_has_term inside
the try/catch. This should not change anything as that function
already traps any Xapian exceptions.
[1]: id:wwuk039sk2p.fsf@chaotikum.eu
|
|
_notmuch_message_delete can return (at least)
NOTMUCH_STATUS_XAPIAN_EXCEPTION, which we should not ignore.
|
|
By sharing the existing logic used by the sexp query parser, this
allows negative lastmod revisions to be interpreted as relative to the
most recent revision.
|
|
This will permit the re-use of the same logic in the infix query
parser. The location of the shared code in the infix side is for
consistency with the other shared parsing logic. It will make more
sense when a Xapian field processor is added for the lastmod prefix.
|
|
Test the relatively trivial logic changes for the sexp query parser
first before refactoring that logic to share with the infix query
parser.
|
|
There is some duplication of code here, but not all of the locations
valid to find a database make sense to create. Furthermore we nead two
passes, so the control flow in _choose_database_path would get a bit
convoluted.
|
|
This simplifies the logic of creating the directory path when it doesn't
exist.
|
|
This makes the error handling available for re-use. Using
g_mkdir_with_parents also handles the case of a pre-existing
directory. This introduces new functionality, namely creating the
parent directories, which will be useful for creating directories like
'.local/share/notmuch/default'.
|
|
Fix the bug reported at [1].
The parameter expansion for regex and wildcard modifiers has to be
done a bit differently, because their arguments are not s-expressions
defining complete Xapian queries.
[1]: id:87o7yxqxy6.fsf@code.pm
|
|
It can be tedious to use "" inside of a string, e.g. in a shell script.
|
|
Support this syntax for constincy with (data from to) ranges.
|
|
The database needs to be writable because the list of stored file
names will change in general.
|
|
In order for a database to actually be writeable, it must be the case that it
is open, not just the correct type of Xapian object. By explicitely
checking, we are able to provide better error reporting, in particular
for the previously broken test in T566-lib-message.
|
|
These were missed when the corresponding status codes were added.
|
|
Make the behaviour when passed NULL consistent with
notmuch_filenames_valid. The library already passes the result of
notmuch_message_get_tags without checking for NULL, so it should be
handled.
|
|
Mainly to propagate information about Xapian exceptions.
|
|
Xapian exceptions are not something that can be ignored, in general.
|
|
In either C++17 (or later) mode, or when running cppcheck, this can be
used to selectively generate warnings about discarded return values.
|
|
This function has been unused since commit 4083fd8.
|
|
Some code movement is needed to make sure the cache is only
invalidated when the Xapian operation succeeds.
|
|
As far as I know, this is just a style / consistency thing, unless
notmuch code starts defining FALSE inconsistently with false.
|
|
Initially reported by Eliza Vasquez [1] (via valgrind).
[1]: id:87o7zxj086.fsf@eliza.
|
|
a1d139de ("lib: add sexp: prefix to Xapian (infix) query parser.",
2022-04-09) introduced sfsexp infix queries. This requires the infix
preprocessor to be built in in a way which does not require sfsexp when
notmuch is built without it.
Make the preprocessor throw a Xapian error in this case (and fix the
build).
Signed-off-by: Michael J Gruber <git@grubix.eu>
|
|
This is analogous to the "infix" prefix provided by the s-expression
based query parser.
|
|
Since Xapian does not preserve quotes when passing the subquery to a
field processor, we have to make a guess as to what the user
intended. Here the added assumption is that a string surrounded by
parens is not intended to be a phrase.
|
|
One new status value and one configuration value added.
|
