TODO: Add some notes to fix some recently noted problems.

[notmuch] / TODO

Fix the things that are causing the most pain to new users
----------------------------------------------------------
1. A new import is tagging all messages as "inbox" -- total pain

2. Allow an easy way to get tags from directory names (if the user has them)

Emacs interface (notmuch.el)
----------------------------
Switch the notmuch-search view to use "notmuch search --format=json"
to fix large classes of bugs regarding poorly-escaped output and lame
regular expressions. (The most recently found, unfixed example is the
sender's name containing ';' which causes emacs to drop a search
result.) This may require removing the outer array from the current
"notmuch search --format=json" results.

Enhance '+' and '-' in the search view to operate on an entire region
if set.

Fix '*' to work by simply calling '+' or '-' on a region consisting of
the entire buffer.

Add a global keybinding table for notmuch, and then view-specific
tables that add to it.
        
Add a '|' binding from the search view.

Add support for choosing from one of the user's configured email
addresses for the From line.

Make 'notmuch-show-pipe-message have a private history.

Add support for a delete keybinding that adds a "deleted" tag to the
current message/thread and make searches not return deleted messages
by default, (unless the user asks explicitly for deleted messags in
the search query).

Add support to "mute" a thread (add a "muted" tag and then don't
display threads in searches by default where any message of the thread
has the "muted" tag).

Fix i-search to open up invisible citations as necessary.

Make '=' count from the end rather than from the beginning if more
than half-way through the buffer.

Fix to automatically wrap long headers (for RFC compliance) before
sending. This should probably just be fixed in message-mode itself,
(but perhaps we can have a notmuch-message-mode that layers this on
top).

Implement Fcc and use it for all messages, (whether a new composition,
a reply, or a forward). This again may require a notmuch-message-mode
that extends message-mode.

Stop hiding the headers so much in the thread-view mode.

Allow opening a message in thread-view mode by clicking on either
line.

Automatically open a message when navigating to it with N or P.

Change 'a' command in thread-view mode to only archive open messages.

Add a binding to open all closed messages.

Make all authors and subjects available to isearch, (hidden by default
but with magic expansion while isearching).

Fix notmuch-hello as follows:

        1. Change the "notmuch" and message count in the welcome sentence to
           not be buttons.

        2. Put the saved searches (if any) before the search bar.

        3. When `notmuch-hello' (or even `notmuch' when it gets its new name)
           is invoked directly, move to the first button, (go to point-min and
           then call widget-forward). That is, if the user has any saved
           searches, then point will be on the first one. If the user has no
           saved searches, then point will be on the search bar.

        4. Fix refresh of notmuch-hello to leave point in the same logical
           place, (the same saved-search widget at least).

Change the 'a'rchive command in the thread view to only archive open
messages.

Emacs saved-search interface
----------------------------
Here's a proposal Carl wrote (id:87einafy4u.fsf@yoom.home.cworth.org):

  So what I'm imagining for the default notmuch view is something like
  this:

          Welcome to notmuch.

              Notmuch search: _________________________________________

          Saved searches:

              55,342      All messages
                  22      Inbox

          Recent searches:

                   1      from:"someone special" and tag:unread
                  34      tag:notmuch and tag:todo

          Click (or press Enter) on any search to see the results.
          Right-click (or press Space) on any recent search to save it.

  So the "saved searches" portion of the view is basically just what
  notmuch-folder displays now. Above that there's an obvious place to
  start a new search, (in a slightly more "web-browser-like" way than the
  typical mini-buffer approach).

  All recent searches appear in the list at the bottom automatically, and
  there's the documented mechanism for saving a search, (giving it a name
  and having it appear above).

Portability
-----------
Fix configure script to test each compiler warning we want to use.

Completion
----------
Fix bash completion to complete multiple search options (both --first
and *then* --max-threads), and also complete value for --sort=
(oldest-first or newest-first).

notmuch command-line tool
-------------------------
Fix the --format=json option to not imply --entire-thread.

Implement "notmuch search --exclude-threads=<search-terms>" to allow
for excluding muted threads, (and any other negative, thread-based
filtering that the user wants to do).

Fix "notmuch show" so that the UI doesn't fail to show a thread that
is visible in a search buffer, but happens to no longer match the
current search. (Perhaps add a --matching=<secondary-search-terms>
option (or similar) to "notmuch show".) For now, this is being worked
around in the emacs interface by noticing that "notmuch show" returns
nothing and re-rerunning the command without the extra arguments.

Teach "notmuch search" to return many different kinds of results. Some
ideas:

        notmuch search --output=threads # Default if no --output is given
        notmuch search --output=messages
        notmuch search --output=tags
        notmuch search --output=addresses
        notmuch search --output=terms

Add a "--format" option to "notmuch search", (something printf-like
for selecting what gets printed).

Add a "--count-only" (or so?) option to "notmuch search" for returning
the count of search results.

Give "notmuch restore" some progress indicator.

Fix "notmuch restore" to operate in a single pass much like "notmuch
dump" does, rather than doing N searches into the database, each
matching 1/N messages.

Add a "-f <filename>" option to select an alternate configuration
file.

Allow configuration for filename patterns that should be ignored when
indexing.

Replace the "notmuch part --part=id" command with "notmuch show
--part=id", (David Edmonson wants to rewrite some of "notmuch show" to
provide more MIME-structure information in its output first).

Replace the "notmuch search-tags" command with "notmuch search
--output=tags".

Fix to avoid this ugly message:

        (process:17197): gmime-CRITICAL **: g_mime_message_get_mime_part: assertion `GMIME_IS_MESSAGE (message)' failed
        Warning: Not indexing empty mime part.

  This probably means adding a test case to generate that message,
  filing an upstream bug against GMime, and then silencing the
  notmuch-generated portion of the warning (so that once GMime is
  fixed, this is all silent).

Simplify notmuch-reply to simply print the headers (we have the
original values) rather than calling GMime (which encodes) and adding
the confusing gmime-filter-headers.c code (which decodes).

notmuch library
---------------
Add an interface to accept a "key" and a byte stream, rather than a
filename.

Provide a sane syntax for date ranges. First, we don't want to require
both endpoints to be specified. For example it would be nice to be
able to say things like "since:2009-01-1" or "until:2009-01-1" and
have the other endpoint be implicit. Second we'd like to support
relative specifications of time such as "since:'2 months ago'". To do
any of this we're probably going to need to break down an write our
own parser for the query string rather than using Xapian's QueryParser
class.

Make failure to read a file (such as a permissions problem) a warning
rather than an error (should be similar to the existing warning for a
non-mail file).

Actually compile and install a libnotmuch shared library.

Fix to use the *last* Message-ID header if multiple such headers are
encountered, (I noticed this is one thing that kept me from seeing the
same message-ID values as sup).

Add support for configuring "virtual tags" which are a tuple of
(tag-name, search-specification). The database is responsible for
ensuring that the virtual tag is always consistent.

Indicate to the user if two files with the same message ID have
content that is actually different in some interesting way. Perhaps
notmuch initially sees all changes as interesting, and quickly learns
from the user which changes are not interesting (such as the very
common mailing-list footer).

Fix notmuch_query_count_messages to share code with
notmuch_query_search_messages rather than duplicating code. (And
consider renaming it as well.)

Provide a mechanism for doing automatic address completion based on
notmuch searches. Here was one proposal made in IRC:

        <cworth> I guess all it would really have to be would be a way
                 to configure a series of searches to try in turn,
                 (presenting ambiguities at a given single level, and
                 advancing to the next level only if one level
                 returned no matches).
        <cworth> So then I might have a series that looks like this:
        <cworth> notmuch search --output=address_from tag:address_book_alias
        <cworth> notmuch search --output=address_to tag:sent
        <cworth> notmuch search --output=address_from
        <cworth> I think I might like that quite a bit.
        <cworth> And then we have a story for an address book for
                 non-emacs users.

Provide a ~me Xapian synonym for all of the user's configured email
addresses.

Add symbol hiding so that we don't risk leaking any private symbols
into the shared-library interface.

Audit all libnotmuch entry points to ensure that all Xapian calls are
wrapped in a try/catch block.

Fix the "count" functionality to be exact as Olly explained in IRC:

        ojwb> cworth: if you set the check_at_least parameter to the
        database size, get_matches_estimated() will be exact

Search syntax
-------------
Implement support for "tag:*" to expand to all tags.

Fix "notmuch search to:" to be less confusing. Many users expect this
to search for all messages with a To: header, but it instead searches
for all messages with the word "to". If we don't provide the first
behavior, perhaps we should exit on an error when a configured prefix
is provided with no value?

Support "*" in all cases and not just as a special case. That is, "* "
should also work, as well as "* and tag:inbox".

Implement a syntax for requesting set-theoertic operations on results
of multiple searches. For example, I would like to do:

        "tag:inbox" SET-SUBTRACT "tag:muted"

    as well as:

        "tag:notmuch and <date-range>" SET-INTERSECT
        "tag:notmuch and not (tag:merged or tag:postponed)"

    See id:3wdpr282yz2.fsf@testarossa.amd.com for more details on the
    use cases of the above.

Database changes
----------------
Store a reference term for every message-id that appears in
References. We just started doing this for newly-added documents, but
at the next convenient database-schema upgrade, we should go back and
fix old messages to be consistent.

Start indexing the List-Id header, (and re-index this header for
existing messages at the next database upgrade).

Start indexing the message file's directory ana make it available for
search as "folder:" (and re-index this value for existing messages at
the next database upgrade).

Add support for the user to specify custom headers to be indexed (and
re-index these for existing messages at the next database upgrade).

Test suite
----------
Start testing --format=json.

Achieve 100% test coverage with the test suite.

Modularize test suite (to be able to run individual tests).

Summarize test results at the end.

Fix the insane quoting nightmare of the test suite, (and once we do
that we can actually test the implicit-phrase search feature such as
"notmuch search 'body search (phrase)'"

Test "notmuch reply" choosing the correct email address from the
Received header when no configured email address appears in To or Cc.

General
-------
Audit everything for dealing with out-of-memory (and drop xutil.c).

Investigate why the notmuch database is slightly larger than the sup
database for the same corpus of email.

Makefile should print message teaching user about LD_LIBRARY_PATH (or
similar) if libdir is not set to a directory examined by ldconfig.

Fix notmuch build system to work with gold, (apt-get
binutils-gold). Need the notmuch binary to explicitly link againsts
libraries it depends on, (rather than just getting them via
libnotmuch).