lei q: parallelize Maildir and mbox writing

With 4 dedicated workers, this seems to provide a 100-120%
speedup on a 4 core machine when writing thousands of search
results to a Maildir or mbox.  This also sets us up for
high-latency IMAP destinations in the future.

This opens the door to more speedup opportunities such
as optimizing dedupe locking and other ways to reduce
contention.

This change is fairly complex and convoluted, unfortunately.
Further work may allow us to simplify it and even improve
performance.

1 files changed, 9 insertions, 3 deletions

diff --git a/lib/PublicInbox/LeiQuery.pm b/lib/PublicInbox/LeiQuery.pm
index a80d5887..d6e801e3 100644
--- a/lib/PublicInbox/LeiQuery.pm
+++ b/lib/PublicInbox/LeiQuery.pm
@@ -41,11 +41,17 @@ sub lei_q {
         $j = 1 if !$opt->{thread};
         $j++ if $opt->{'local'}; # for sto->search below
         $self->atfork_prepare_wq($lxs);
-        $lxs->wq_workers_start('lei_xsearch', $j, $self->oldset)
-                // $lxs->wq_workers($j);
+        $lxs->wq_workers_start('lei_xsearch', $j, $self->oldset);
+        $self->{lxs} = $lxs;
 
-        # no forking workers after this
         my $ovv = PublicInbox::LeiOverview->new($self) or return;
+        if (my $l2m = $self->{l2m}) {
+                $j = 4 if $j <= 4; # TODO configurable
+                $self->atfork_prepare_wq($l2m);
+                $l2m->wq_workers_start('lei2mail', $j, $self->oldset);
+        }
+
+        # no forking workers after this
         my $sto = $self->_lei_store(1);
         unshift(@srcs, $sto->search) if $opt->{'local'};
         my %mset_opt = map { $_ => $opt->{$_} } qw(thread limit offset);