From: wmorgan <wmorgan@5c8cc53c-5e98-4d25-b20a-d8db53a31250>
Date: Wed, 21 Feb 2007 03:58:16 +0000 (+0000)
Subject: added maildir support (finally!)
X-Git-Url: https://git.notmuchmail.org/git?a=commitdiff_plain;h=3206b4b3d29a3df3d32f70aa3258d43b6cece346;p=sup

added maildir support (finally!)

git-svn-id: svn://rubyforge.org/var/svn/sup/trunk@339 5c8cc53c-5e98-4d25-b20a-d8db53a31250
---

diff --git a/History.txt b/History.txt
index ac5e1ba..723b6d8 100644
--- a/History.txt
+++ b/History.txt
@@ -1,3 +1,7 @@
+== 0.0.8 / XXXX
+* Maildir support.
+* More bugfixes. Will they ever end?
+
 == 0.0.7 / 2007-02-12
 * Split sup-import into two bits: sup-import and sup-add.
 * Command-line arguments now handled by trollop.
diff --git a/Manifest.txt b/Manifest.txt
index c4b9da6..0f33838 100644
--- a/Manifest.txt
+++ b/Manifest.txt
@@ -23,6 +23,7 @@ lib/sup/index.rb
 lib/sup/keymap.rb
 lib/sup/label.rb
 lib/sup/logger.rb
+lib/sup/maildir.rb
 lib/sup/mbox.rb
 lib/sup/mbox/loader.rb
 lib/sup/mbox/ssh-file.rb
diff --git a/README.txt b/README.txt
index bc271fa..748cb91 100644
--- a/README.txt
+++ b/README.txt
@@ -11,9 +11,9 @@ with the speed and simplicity of a console interface.
 Sup makes it easy to:
 - Handle massive amounts of email.
 
-- Mix email from different sources: mbox files (even across
-  different machines), IMAP folders, POP accounts, and GMail
-  accounts.
+- Mix email from different sources: mbox files (even across different
+  machines), Maildir directories, IMAP folders, POP accounts, and
+  GMail accounts.
 
 - Instantaneously search over your entire email collection. Search
   over body text, or use a query language to combine search
@@ -40,7 +40,7 @@ Features:
   operability, regardless of how much amount of email you have.
 
 - Immediate full-text search of your entire email archive, using the
-  Ferret query langauge. Search over message bodies, labels, from: and
+  Ferret query language. Search over message bodies, labels, from: and
   to: fields, or any combination thereof.
 
 - Thread-centrism. Operations are performed at the thread, not the
@@ -109,21 +109,24 @@ Current limitations which will be fixed:
 * ncurses
 * rmail
 * highline
+* trollop
+* net-ssh
 
 == INSTALL:
 
 * gem install sup -y
 
 == KNOWN BUGS IN OTHER PACKAGES:
+
 * If you get an error about frozen strings in RubyMail when importing
   certain messages with attachments, in rmail, change line 159 of
   multipart.rb to:
     chunk = chunk[0..start]
+  This is because RubyMail hasn't been updated since like Ruby 1.8.2.
+  Please bug Matt Lickey.
 * Occasionally Ferret produces something the Ruby GC doesn't like
   (particularly when importing messages from very large sources).
   No worries, just re-run sup-import. (This is unresolved atm.)
-* There are a couple other Ferret issues with outstanding patches but
-  they are pretty rare.
 * If you are using IMAP or Maildir and see this error:
     /usr/local/lib/ruby/1.8/yaml.rb:133:in `transfer': allocator undefined for Bignum (TypeError)
   then you need to upgrade to Ruby 1.8.5. YAML in earlier versions
diff --git a/bin/sup-add b/bin/sup-add
index 69749bb..b0d8457 100644
--- a/bin/sup-add
+++ b/bin/sup-add
@@ -6,8 +6,6 @@ require 'highline/import'
 require 'trollop'
 require "sup"
 
-Thread.abort_on_exception = true # make debugging possible
-
 $opts = Trollop::options do
   version "sup-add (sup #{Redwood::VERSION})"
   banner <<EOS
@@ -32,6 +30,9 @@ For IMAP folders, use the form (note no username or password!):
     imaps://<machine name>/         # secure, "INBOX" folder
     imaps://<machine name>/<folder> # secure, arbitrary folder 
 
+For Maildir folders, use the form:
+    maildir://<path to Maildir directory>
+
 Options are:
 EOS
   opt :archive, "Automatically archive all new messages from these sources."
@@ -80,22 +81,30 @@ index.load
 
 ARGV.each do |uri|
   uri = "mbox://#{uri}" unless uri =~ %r!://!
+
   if !$opts[:force_new] && index.source_for(uri) 
     say "Already know about #{uri}; skipping."
     next
   end
+
+  parsed_uri = URI(uri)
+
   source = 
-    case uri
-    when %r!^mbox\+ssh://!
+    case parsed_uri.scheme
+    when "mbox+ssh"
       say "For SSH connections, if you will use public key authentication, you may leave the username and password blank."
       say ""
       username, password = get_login_info uri, index.sources
-      Redwood::MBox::SSHLoader.new(uri, username, password, nil, !$opts[:unusual], $opts[:archive])
-    when %r!^imaps?://!
+      Redwood::MBox::SSHLoader.new uri, username, password, nil, !$opts[:unusual], $opts[:archive]
+    when "imap", "imaps"
       username, password = get_login_info uri, index.sources
-      Redwood::IMAP.new(uri, username, password, nil, !$opts[:unusual], $opts[:archive])
+      Redwood::IMAP.new uri, username, password, nil, !$opts[:unusual], $opts[:archive]
+    when "maildir"
+      Redwood::Maildir.new uri, nil, !$opts[:unusual], $opts[:archive]
+    when "mbox"
+      Redwood::MBox::Loader.new uri, nil, !$opts[:unusual], $opts[:archive]
     else
-      Redwood::MBox::Loader.new(uri, nil, !$opts[:unusual], $opts[:archive])
+      Trollop::die "Unknown source type #{parsed_uri.scheme.inspect}"      
     end
   say "Adding #{source}..."
   index.add_source source
diff --git a/bin/sup-import b/bin/sup-import
index cf2b41a..c0c635c 100644
--- a/bin/sup-import
+++ b/bin/sup-import
@@ -5,8 +5,6 @@ require 'rubygems'
 require 'trollop'
 require "sup"
 
-Thread.abort_on_exception = true # make debugging possible
-
 class Float
   def to_s; sprintf '%.2f', self; end
 end
diff --git a/doc/TODO b/doc/TODO
index db9b7d5..e2bda88 100644
--- a/doc/TODO
+++ b/doc/TODO
@@ -1,13 +1,13 @@
 for 0.0.8
 ---------
-bugfix: when one new message comes into an imap folder, we don't catch
-   it until a restart
-bugfix: triggering a pageup when cursor scrolling up jumps to the bottom of the
-  page rather than the next line
-create attachments
-forward attachments
-warnings: top-posting, missing attachment
-maildir
+x maildir
+_ bugfix: when one new message comes into an imap folder, we don't
+   catch it until a restart
+_ bugfix: triggering a pageup when cursor scrolling up jumps to the
+   bottom of the page rather than the next line
+_ create attachments
+_ forward attachments
+_ warnings: top-posting, missing attachment
 
 for 0.0.9
 ---------
diff --git a/lib/sup.rb b/lib/sup.rb
index 2df5e52..76b435d 100644
--- a/lib/sup.rb
+++ b/lib/sup.rb
@@ -127,6 +127,7 @@ require "sup/update"
 require "sup/message"
 require "sup/source"
 require "sup/mbox"
+require "sup/maildir"
 require "sup/imap"
 require "sup/person"
 require "sup/account"
diff --git a/lib/sup/maildir.rb b/lib/sup/maildir.rb
new file mode 100644
index 0000000..f15befc
--- /dev/null
+++ b/lib/sup/maildir.rb
@@ -0,0 +1,124 @@
+require 'rmail'
+require 'uri'
+
+module Redwood
+
+## Maildir doesn't provide an ordered unique id, which is what Sup
+## requires to be really useful. So we must maintain, in memory, a
+## mapping between Sup "ids" (timestamps, essentially) and the
+## pathnames on disk.
+
+class Maildir < Source
+  SCAN_INTERVAL = 30 # seconds
+
+  def initialize uri, last_date=nil, usual=true, archived=false, id=nil
+    super
+
+    @dir = URI(uri).path
+    @ids = []
+    @ids_to_fns = {}
+    @labels = [:unread]
+    @last_scan = nil
+    @mutex = Mutex.new
+  end
+
+  def load_header id
+    scan_mailbox
+    with_file_for(id) { |f| MBox::read_header f }
+  end
+
+  def load_message id
+    scan_mailbox
+    with_file_for(id) { |f| RMail::Parser.read f }
+  end
+
+  def raw_header id
+    scan_mailbox
+    ret = ""
+    with_file_for(id) do |f|
+      until f.eof? || (l = f.gets) =~ /^$/
+        ret += l
+      end
+    end
+    ret
+  end
+
+  def raw_full_message id
+    scan_mailbox
+    with_file_for(id) { |f| f.readlines.join }
+  end
+
+  def scan_mailbox
+    return if @last_scan && (Time.now - @last_scan) < SCAN_INTERVAL
+
+    cdir = File.join(@dir, 'cur')
+    ndir = File.join(@dir, 'cur')
+
+    begin
+      @ids, @ids_to_fns = @mutex.synchronize do
+        ids, ids_to_fns = [], {}
+        (Dir[File.join(cdir, "*")] + Dir[File.join(ndir, "*")]).map do |fn|
+          id = make_id fn
+          ids << id
+          ids_to_fns[id] = fn
+        end
+        [ids.sort, ids_to_fns]
+      end
+    rescue SystemCallError => e
+      die "Problem scanning Maildir directories: #{e.message}."
+    end
+    
+    @last_scan = Time.now
+  end
+
+  def each
+    scan_mailbox
+    start = @ids.index(cur_offset || start_offset) or die "Unknown message id #{cur_offset || start_offset}.", :suggest_rebuild => true # couldn't find the most recent email
+
+    start.upto(@ids.length - 1) do |i|         
+      id = @ids[i]
+      self.cur_offset = id
+      yield id, @labels.clone
+    end
+  end
+
+  def start_offset
+    scan_mailbox
+    @ids.first
+  end
+
+  def end_offset
+    scan_mailbox
+    @ids.last
+  end
+
+  def pct_done; 100.0 * (@ids.index(cur_offset) || 0).to_f / (@ids.length - 1).to_f; end
+
+private
+
+  def die message, opts={}
+    message += " It is likely that messages have been deleted from this Maildir mailbox. Please run sup-import --rebuild #{to_s} to correct this problem." if opts[:suggest_rebuild]
+    self.broken_msg = message
+    Redwood::log message
+    BufferManager.flash "Error communicating with Maildir. See log for details." if BufferManager.instantiated?
+    raise SourceError, message
+  end
+  
+  def make_id fn
+    # use 7 digits for the size. why 7? seems nice.
+    sprintf("%d%07d", File.ctime(fn), File.size(fn)).to_i
+  end
+
+  def with_file_for id
+    fn = @ids_to_fns[id] or die "No such id: #{id.inspect}.", :suggest_rebuild => true
+    begin
+      File.open(fn) { |f| yield f }
+    rescue SystemCallError => e
+      die "Problem reading file for id #{id.inspect}: #{fn.inspect}: #{e.message}."
+    end
+  end
+end
+
+Redwood::register_yaml(Maildir, %w(uri cur_offset usual archived id))
+
+end
diff --git a/www/index.html b/www/index.html
index 1d590bb..8aba073 100644
--- a/www/index.html
+++ b/www/index.html
@@ -56,10 +56,11 @@ the <a href="Philosophy.txt">philosophical statement</a>.
 
 <h2>Status</h2>
 
-<p> The current version of Sup is 0.0.7, released February 12th, 2007.
+<p> The current version of Sup is 0.0.8, released XXXX.
     This is a beta release. It is unix-centric and has no i18n
-    support. It supports only mbox, mbox+ssh, and IMAP, not POP,
-    GMail or Maildir. I plan to fix all of these problems. </p>
+    support. It supports mbox, mbox over ssh, IMAP, and Maildir.  It
+    does not support POP or GMail. I plan to fix all of these
+    problems. </p>
 
 <p> Other than those limitations, it works great! I use it for my
     everyday email, and it makes dealing with 85,000 messages a