David Bremner [Sat, 9 Jan 2021 11:29:07 +0000 (07:29 -0400)]
CLI/insert: support split database and mail root
The new test is in T055-path-config because it uses the helper
function split_config, and because it seems easier to put the
database path related tests in one place.
David Bremner [Mon, 4 Jan 2021 00:04:46 +0000 (20:04 -0400)]
lib/open: support XDG_DATA_HOME as a fallback database location.
This changes some error reporting, either intentionally by reporting
the highest level missing directory, or by side effect from looking in
XDG locations when given null database location.
David Bremner [Sat, 9 Jan 2021 11:25:25 +0000 (07:25 -0400)]
CLI/new: support split database and mail location
This adds new state variable for the mail root, and uses it most
places db_path was used. The notable exception is dumps from
backups. The latter will be dealt with properly in a future commit.
David Bremner [Wed, 17 Feb 2021 00:08:23 +0000 (20:08 -0400)]
lib/open: check for split configuration when creating database.
The main functionality will be tested when notmuch-new is converted to
support split configuration. Here only the somewhat odd case of split
mail root which is actually symlinked to the database path is tested.
David Bremner [Thu, 31 Dec 2020 22:20:31 +0000 (18:20 -0400)]
lib: support splitting mail from database location.
Introduce a new configuration value for the mail root, and use it to
locate mail messages in preference to the database.path (which
previously implied the mail messages were also in this location.
Initially only a subset of the CLI is tested in a split
configuration. Further changes will be needed for the remainder of the
CLI to work in split configurations.
David Bremner [Sat, 30 Jan 2021 17:57:32 +0000 (13:57 -0400)]
lib/open: factor out the second half of n_d_open_with_config
The idea is to allow reuse in n_d_create_with_config. This is
primarily code movement, with some changes in error messages to reduce
the number of input parameters.
David Bremner [Fri, 29 Jan 2021 11:44:18 +0000 (07:44 -0400)]
lib/open: factor out library initialization
This is slightly more tidy, but more importantly it allows for re-use
of this code in n_d_create_with_config. That re-use will be crucial
when we no longer call n_d_open_with_config from
n_d_create_with_config.
David Bremner [Tue, 2 Feb 2021 20:43:58 +0000 (16:43 -0400)]
lib: remove "path" from notmuch struct
This removes duplication between the struct element and the
configuration string_map entry. Create a simple wrapper for setting
the database path that makes sure the trailing / is stripped.
David Bremner [Sat, 6 Feb 2021 22:50:20 +0000 (18:50 -0400)]
CLI/show: complete conversion to new configuration framework.
In order to open the database in main() for this command, we may need
to re-open it in the (possibly less common) case where crypto options
require write access.
David Bremner [Fri, 19 Mar 2021 01:58:02 +0000 (22:58 -0300)]
CLI/new: drop the write lock to run the pre-new hook.
This fixes a bug reported in [1]. In principle it could be possible
avoid one of these reopens, but it complicates the logic in main with
respect to creating new databases.
David Bremner [Fri, 19 Mar 2021 01:54:13 +0000 (22:54 -0300)]
test: Add tests for write access to database from hooks.
Recent changes to configuration handling meant the pre-new hook was
run while the database was open read only, limiting what could be done
in the hook. Add some known broken tests for this problem, as well as
a regression test for write access from the post-new hook.
David Bremner [Sun, 14 Mar 2021 14:01:58 +0000 (11:01 -0300)]
lib: save path of xapian database in notmuch struct.
This will allow re-opening in a different mode (read/write
vs. read-only) with current Xapian API. It will also prove useful when
updating the compact functions to support more flexible database
location.
David Bremner [Sun, 14 Mar 2021 14:01:57 +0000 (11:01 -0300)]
lib: publish API for notmuch_database_reopen
Include the (currently unused) mode argument which will specify which
mode to re-open the database in. Functionality and docs to be
finalized in a followup commit.
Tomi Ollila [Thu, 25 Feb 2021 20:04:30 +0000 (22:04 +0200)]
test: T020-compact.sh: fix work directory location
Fix use of $TEST_DIRECTORY ($NOTMUCH_BUILDDIR/test/) with use of
$TMP_DIRECTORY ($NOTMUCH_BUILDDIR/test/tmp.T020-compact/ in case
of T020-compact.sh) as root directory where to write test files
and directories.
David Bremner [Mon, 15 Feb 2021 00:55:55 +0000 (20:55 -0400)]
test: Fix race condition in T568-lib-thread.sh
The assignment of thread-ids is (apparently) non-deterministic in a
way that mostly seems to show up on multicore machines. In my tests
the number is different from that previously assumed by this test
about 15% of the time on a 50 thread (25 core) Xeon.
Since message id's are fixed, use a message known to be in the thread
of interest to pick out the correct thread-id.
David Bremner [Mon, 15 Feb 2021 20:43:50 +0000 (16:43 -0400)]
fix build failure with glib 2.67
Based on a patch from Michael J Gruber [1]. As of glib 2.67 (more
specifically [2]), including "gmime-extra.h" inside an extern "C"
block causes build failures, because glib is using C++ features.
Observing that "gmime-extra.h" is no longer needed in
notmuch-private.h, which can simply delete that include, but
we have to correspondingly move the includes which might include
it (in particular crypto.h) out of the extern "C" block also.
This seems less fragile than only moving gmime-extra, and relying on
preprocessor sentinels to keep the deeper includes from happening.
Move to the include to the outside of the extern block.
This is the last bit of "python" left in the notmuch codebase.
https://www.python.org/dev/peps/pep-0394/#recommendation encourages
"third-party distributors" to use more-specific shebang lines. I'm
not certain that the notmuch project itself is a "third-party
contributor" but I think this is a safe way to encourage people to use
python3 when they're developing notmuch.
We already have python3 explicitly elsewhere in the codebase for
developers (in nmbug).
Signed-off-by: Daniel Kahn Gillmor <dkg@fifthhorseman.net>
David Bremner [Sun, 10 Jan 2021 15:54:35 +0000 (11:54 -0400)]
doc: describe new config framework
Remove STORED IN DATABASE discussion, describe search rules.
Currently profiles seem a bit pointless. They will make more sense
when they apply to databases as well.
David Bremner [Sun, 10 Jan 2021 11:50:14 +0000 (07:50 -0400)]
CLI: use configured hook directory
This enables support for hooks outside the database directory.
It relies strongly on configuration information being usable between
closing the database and destroying it.
David Bremner [Sat, 9 Jan 2021 13:19:50 +0000 (09:19 -0400)]
lib/config: add HOOK_DIR
The hook directory configuration needs to be kept in synch with the
other configuration information, so add scaffolding to support this at
database opening time.
David Bremner [Sat, 2 Jan 2021 17:10:39 +0000 (13:10 -0400)]
cli/new: convert to new config framework
In addition to the same type of changes as converting other
subcommands, add the possibility of creating a database at the top
level. It would probably make sense to use this for insert as well.
David Bremner [Sun, 3 Jan 2021 12:30:46 +0000 (08:30 -0400)]
lib: add NOTMUCH_STATUS_NO_CONFIG
This will allow client code to provide more meaningful diagnostics. In
particular it will enable "notmuch new" to continue suggsting the user
run "notmuch setup" to create a config after "notmuch new" is
transitioned to the new configuration framework.
David Bremner [Sat, 8 Aug 2020 14:16:53 +0000 (11:16 -0300)]
CLI/show: mostly switch show to new config framework
This will need some cleanup when the transition completes, and we stop
passing notmuch_config_t structs to the subcommands.
Unlike the general case, we open the database in the subcommand, since
we don't know whether it should be opened read/write until we parse
the command line arguments.
Add a test to make sure passing config file on the command line is not
broken by these or future config related changes.
David Bremner [Sat, 8 Aug 2020 14:16:51 +0000 (11:16 -0300)]
cli/config: add accessor for config file name
This is intended for use in temporary code transitioning to the new
configuration system. The name is chosen to avoid cluttering the
notmuch_config_* namespace further with non-library functions.
David Bremner [Wed, 26 Aug 2020 11:43:33 +0000 (08:43 -0300)]
CLI/insert: convert to new config framework.
The new talloc context is needed to run the hook at the very end of
the function. That in turn is needed so that this process gives up the
write lock on the database.
David Bremner [Wed, 26 Aug 2020 10:54:10 +0000 (07:54 -0300)]
cli/dump: convert to new config framework
This conversion is trivial because the only configuration information
accessed by dump is that stored in the database (in order to dump
it). We do need to be careful to keep the write lock on the database
to ensure dump consistency.
David Bremner [Tue, 22 Dec 2020 12:13:51 +0000 (08:13 -0400)]
CLI/count: switch to new configuration framework
The main effort is changing from the old argv style config list
iterators to the new more opaque ones provided by the library (and
backed by the database+file config cache).
David Bremner [Mon, 21 Dec 2020 12:55:14 +0000 (08:55 -0400)]
CLI: add (unused) database argument to subcommands.
This will allow transitioning individual subcommands to the new
configuration framework. Eventually when they are all converted we can
remove the notmuch_config_t * argument.
For now, live with the parameter shadowing in some some subcommands;
it will go away when they are converted.
David Bremner [Wed, 26 Aug 2020 01:36:43 +0000 (22:36 -0300)]
lib/config: add notmuch_config_key_{get,set}
By using an enum we can have better error detection than copy pasting
key strings around.
The question of what layer this belongs in is a bit
tricky. Historically most of the keys are defined by the CLI. On the
other hand features like excludes are supported in the
library/bindings, and it makes sense to configure them from the
library as well.
The somewhat long prefix for notmuch_config_t is to avoid collisions
with the existing usage in notmuch-client.h.
David Bremner [Mon, 21 Dec 2020 16:12:09 +0000 (12:12 -0400)]
CLI: generalize notmuch_config_mode_t
The renaming and extra values will make sense when we start to convert
subcommands to the new configuration framework. It will also avoid
collisions with a new enum for configuration keys to be introduced in
a future commit.
David Bremner [Sat, 8 Aug 2020 14:16:46 +0000 (11:16 -0300)]
lib/open: add support for config profiles and default locations
Fill in the remainder of the documented functionality for
n_d_open_with_config with respect to config file location. Similar
searching default locations of the database file still needs to be
added.
David Bremner [Sat, 8 Aug 2020 14:16:37 +0000 (11:16 -0300)]
lib: cache configuration information from database
The main goal is to allow configuration information to be temporarily
overridden by a separate config file. That will require further
changes not in this commit.
The performance impact is unclear, and will depend on the balance
between number of queries and number of distinct metadata items read
on the first call to n_d_get_config.
David Edmondson [Wed, 3 Feb 2021 09:10:20 +0000 (09:10 +0000)]
emacs: When completing tags, offer each tag once
When prompting for one or more tags to add or remove to/from one or
more threads, ensure that the set of tags offered for completion
contains no duplicates.
Some completion packages (e.g. selectrum) will include every member of
the offered list, resulting in the same tag being indicated as a
possibility several times.
David Bremner [Sun, 17 Jan 2021 12:35:38 +0000 (08:35 -0400)]
test: add (back) upgrade tests
In ee897cab8b721 the upgrade tests from pre v3 databases were
removed. The reasons for that are still valid, but we should still
test the code paths that do the upgrade, and it is relatively
straightforward to do that for v3 to v3 upgrades.
Jonas Bernoulli [Sun, 10 Jan 2021 18:47:22 +0000 (19:47 +0100)]
emacs: avoid type errors due to nil as content-type
The output of "notmuch show --format=sexp --format-version=4"
may contain `:content-type' entries with `nil' as the value,
when it fails to detect the correct value. Account for that
in a few places where we would otherwise risk a type error.
Note that `string=' does not choke on `nil' because it uses
the `symbol-name' when encountering a symbol.
Jonas Bernoulli [Sun, 10 Jan 2021 14:01:12 +0000 (15:01 +0100)]
emacs: notmuch-address-expand-name: use the actual initial-input
Users may type some text into the buffer on an address line, before
actually invoking address completion. We now use that text as the
initial input when we begin address completion.
Previously we did knowingly replace the actual initial input with some
completion candidate that happens to match. Which candidate is used is
essentially random, at least when the actual initial input is short.
As a result users very often had to begin completion by deleting the
less than helpful "initial input".
Jonas Bernoulli [Sun, 10 Jan 2021 14:01:11 +0000 (15:01 +0100)]
emacs: allow opting out of notmuch's address completion
IMO Notmuch should not override the default completion mechanism by
default, at least not globally. But since users are already used to
this behavior it is probably too late to change it. Do the next best
thing and at least allow users to opt out.
Jonas Bernoulli [Sun, 10 Jan 2021 14:01:10 +0000 (15:01 +0100)]
emacs: notmuch-tree-get-match: No longer define as command
When called from code, then this function returns non-nil when the
message at point is a matched message. However it does nothing at all
to present that information to the user when it called interactively.
It is therefore safe to conclude that nobody is using this as a
command.
Jonas Bernoulli [Sun, 10 Jan 2021 14:01:08 +0000 (15:01 +0100)]
emacs: make subr-x available in all libraries
Like `cl-lib' and `pcase', which are already available in all
libraries, `subr-x' also provided many useful functions that
we would like to use.
Making `subr-x' available in every library from the get-go means
that we can use the functions it defines without having to double
check every single time, whether the feature is already available
in the current library.
Jonas Bernoulli [Sun, 10 Jan 2021 14:01:07 +0000 (15:01 +0100)]
emacs: improve how cl-lib and pcase are required
We need to load `cl-lib' at run-time because we use more from it than
just macros. Never-the-less many, but not all libraries required it
only at compile-time, which we got away with because at least some
libraries already required it at run-time as well.
We use `cl-lib' and (currently to a lesser extend) `pcase' throughout
the code-base, which means that we should require these features in
most libraries.
In the past we tried to only require these features in just the
libraries that actually need them, without fully succeeding. We did
not succeed in doing so because that means we would have to check
every time that we use a function from these features whether they
are already being required in the current library.
An alternative would be to add the `require' forms at the top of every
library but that is a bit annoying too.
In order to make sure that these features are loaded when needed but
also to keep the noise down we only require them in "notmuch-lib.el",
which most other libraries require, and in most of the few libraries
that do not do so, namely "notmuch-draft.el", "notmuch-message.el" and
"notmuch-parser.el". ("coolj.el", "make-deps.el", various generated
libraries, and "notmuch-compat.el" are left touched.)
Jonas Bernoulli [Sun, 10 Jan 2021 14:01:06 +0000 (15:01 +0100)]
emacs: avoid unnecessary let-bindings
To some extend this is a personal preference, but the preference is
strongly dependent on whether one is used to a language that makes it
necessary to use variables like this.
This makes it perfectly clear that we are first getting and then using
a "foo":
(use-foo (get-foo))
Sure this has to be read "inside out", but that's something one better
gets used to quickly when dealing with lisp. I don't understand why
one would want to write this instead:
(let ((the-foo (get-foo)))
(use-foo the-foo))
Both `get-foo' and `use-foo' are named in a way that make it very
clear that we are dealing with a "foo". Storing the value in an
additional variable `the-foo' does not make this any more clear.
On the contrary I makes the reader wonder why the author choose to
use a variable. Is the value used more than once? Is the value
being retrieved in one context and then used in another (e.g. when
the current buffer changes)?
Jonas Bernoulli [Sun, 10 Jan 2021 14:01:04 +0000 (15:01 +0100)]
emacs: notmuch-search-stash-thread-id: use notmuch-search-query-string
No longer use the function `notmuch-search-get-query', which does
nothing but return the value of that variable. That function was
added in [1: f47eeac0] for use in `notmuch-read-query' along-side
related `notmuch-show-get-query' and `notmuch-tree-get-query' but
using it here makes little sense.