| Age | Commit message (Collapse) | Author |
|---|
|
|
|
One thread_id value may have multiple thread IDs in it so we need
to separate them out before inserting into our hash.
|
|
Thanks to valgrind for spotting this one.
|
|
Instead of using the recursive "foreach" method, we implement our
own recursive function. This allows us to ignore the signature
component of a multipart/signed message, (which we certainly
don't need to index).
|
|
Ignoring this whitespace seems like a good idea to me, but it's
interfering with my comparisons with sup since sup doesn't do this.
This might be a commit worth dropping in the future since it exists
only for pedantic consistency with sup and not for any reason of its
own.
|
|
We now emit one term per thread_id, rather than the comma-separated
super-term we were doing previously.
|
|
If there's no name given, take the portion of the email addres
before the '@' sign.
One step closer to matching sup's terms in the database.
|
|
Here's another instance where I "knew" gmime must have support for
some functionality, but not finding it, I rolled my own. Now that
I found g_mime_references_decode I'm glad to drop my ugly code.
|
|
This cleans up some old code that was very ugly, (separately opening
the mail file and seeking to the end of the headers to parse the
body). I knew gmime must have had support for transparently decoding
mime content, but I just couldn't find it previously.
Note: Multipart and MultipartSigned parts are not handled yet.
Things are quite happy now. The few differences I see with sup are:
1. sup forces email address domains to lowercase, (I don't think I care)
2. sup and notmuch disagree on ordering of multiple thread_id values
(another thing that's of no concern)
We are still doing one thing wrong when a message belongs to multiple
threads. We've got a nice comma-separated thread-value just like sup,
but then we're also putting in a comma-separated thread-term where
sup does multiple thread terms. That should be an easy fix.
Beyond that, sup and notmuch are still disagreeing on the term lists
for some messages, (I think attachment vs. inline content-disposition
is at least one piece of this). But there are likley still differences
in the heuristics for which chunks of the message body to index. I'll
be looking into this more.
|
|
This provides the thread_id linkage for when a child message is
indexed before the parent.
|
|
The abuse of the generic "value" name was getting very hard to read.
|
|
Currently we're looking up all parents (based on In-reply-to and
References header) and using the list of all thread_id values
from those as our thread_id value. We're missing one step which
sup does which is to also look up any children in the database
that have reference our message ID. So we'll need to do that next.
|
|
This is in preparation for doing a couple of passes over the references,
(one to add terms to the database, and a second to find the thread_id).
We also now parse the In-reply-to header which we were missing before.
We treat it identically to the References header.
|
|
Getting more sup-compatible all the time.
|
|
It's obviously an innocent-enough message, and the right thing is
so easy to do.
|
|
This allows for indexing an arbitrary number of messages with a
single invocation rather than just a single message on the command
line.
|
|
This is a step toward having a program that will index many messages
with a single invocation.
|
|
We're basically matching sup now! (As long as one uses sup with my
special notmuch_index.rb file).
|
|
Pull the "constant" source_id value out from among several calls
that set a value based on the Message ID.
|
|
Getting closer to sup results all the time.
|
|
We identify it based on a trailing ':' on the line before a quote
begins.
At this point the database-dump diff between sup and notmuch is
getting very, very small, (at least for our one test message).
|
|
At this point, we're achieving a result that is *very* close to
what sup does. The only difference is that we are still indexing
the "excerpts from message ..." line, and we are not yet indexing
references.
|
|
This one is complex enough to deserve its own treament.
|
|
Most of this code is fairly clean and works well. One part is
fairly painful---namely extracting the body of an email message
from libgmime. Currently, I'm just extracting the offset to
the end of the headers, and then separately opening the message.
Surely there's a better way.
Anyway, with that the results are looking very similar to sup-sync
now, (as verified by xapian-dump). The only substantial difference
I'm seeing now is that sup does not seem to index quoted portions
of messages nor signatures. I'm not actually sure whether I want
to follow sup's lead in that or not.
|
|
In preparation for actually creating a Xapian index from the
message, (not that we're doing that quite yet).
|
|
Just to make it easier to visually identify where one document ends
and the next begins.
|
|
At the same time, I've started hacking up sup with a new NotmuchIndex
class in the place of the previous XapianIndex class. The new class
stores only the source_info field in the document data, (rather than
a serialized ruby hash with a bunch of data that can be found in the
original message).
Eventually, I plan to replace source_info with a relative filename for
the message, (or even a list of filenames for when multiple messages
in the database share a common message ID).
|
|
The interface for this is cheesy, (bare integer value numbers on the
command line indicating that unserialization is desired for those
value numbers). But this at least lets us print sup databases with
human-readable output for the date values.
|
|
|
|
|
|
|
|
It's not a complete tool yet, but it at least does something now.
|
|
Compiling with -Wall considered useful.
|
|
This will (when it is finished) make a much more reliable way to
ensure that notmuch's sync program behaves identically to sup-sync.
It doesn't actually do anything yet.
|
|
What I've done here is to instrument sup-sync to print the text
and terms objects it constructs just before indexing a message.
Then I've made my g_mime_test program achieve (nearly) identical
output for an example email message, (just missing the body
text). Next we can start shoving this data into a Xapian index.
|
|
Basically just playing with some simple code using libgmime to parse
an email message.
|
