lei index: new command to index mail w/o git storage

Since completely purging blobs from git is slow, users may wish
to index messages in Maildirs (and eventually other local
storage) without storing data in git.

Much code from LeiImport and LeiInput is reused, and a new dummy
FakeImport class supplies a non-storing $im->add and minimize
changes to LeiStore.

The tricky part of this command is to support "lei import"
after a message has gone through "lei index".  Relying on
$smsg->{bytes} == 0 (as we do for external-only vmd storage)
does not work here, since it would break searching for "z:"
byte-ranges when not using externals.

This eventually required PublicInbox::Import::add to use a
SharedKV to keep track of imported blobs and prevent
duplication.

1 files changed, 10 insertions, 5 deletions

diff --git a/lib/PublicInbox/LeiImport.pm b/lib/PublicInbox/LeiImport.pm
index 6a57df47..55925cc5 100644
--- a/lib/PublicInbox/LeiImport.pm
+++ b/lib/PublicInbox/LeiImport.pm
@@ -38,21 +38,20 @@ sub input_maildir_cb { # maildir_each_eml cb
                         warn "E: $f was not from a Maildir?\n";
                 }
         }
-        input_eml_cb($self, $eml, $vmd);
+        $self->input_eml_cb($eml, $vmd);
 }
 
 sub input_net_cb { # imap_each / nntp_each
         my ($url, $uid, $kw, $eml, $self) = @_;
         my $vmd = $self->{-import_kw} ? { kw => $kw } : undef;
         $vmd->{sync_info} = [ $url, $uid ] if $self->{-mail_sync};
-        input_eml_cb($self, $eml, $vmd);
+        $self->input_eml_cb($eml, $vmd);
 }
 
-sub lei_import { # the main "lei import" method
-        my ($lei, @inputs) = @_;
+sub do_import_index ($$@) {
+        my ($self, $lei, @inputs) = @_;
         my $sto = $lei->_lei_store(1);
         $sto->write_prepare($lei);
-        my $self = bless {}, __PACKAGE__;
         $self->{-import_kw} = $lei->{opt}->{kw} // 1;
         my $vmd_mod = $self->vmd_mod_extract(\@inputs);
         return $lei->fail(join("\n", @{$vmd_mod->{err}})) if $vmd_mod->{err};
@@ -83,6 +82,12 @@ sub lei_import { # the main "lei import" method
         $op_c->op_wait_event($ops);
 }
 
+sub lei_import { # the main "lei import" method
+        my ($lei, @inputs) = @_;
+        my $self = bless {}, __PACKAGE__;
+        do_import_index($self, $lei, @inputs);
+}
+
 sub _complete_import {
         my ($lei, @argv) = @_;
         my $sto = $lei->_lei_store or return;