about summary refs log tree commit homepage
path: root/lib/PublicInbox/MboxReader.pm
DateCommit message (Collapse)
2023-11-03treewide: use ->close to call ProcessIO->CLOSE
This will open the door for us to drop `tie' usage from ProcessIO completely in favor of OO method dispatch. While OO method dispatches (e.g. `$fh->close') are slower than normal subroutine calls, it hardly matters in this case since process teardown is a fairly rare operation and we continue to use `close($fh)' for Maildir writes.
2023-10-11treewide: consolidate "From " line removal
Aside from our prior import bugs (fixed in a0c07cba0e5d8b6a (mda: drop leading "From " lines again, 2016-06-26)), we'll always have to be dealing with mutt piping messages to us and `git format-patch' output. So just share the regexp so we can use it everywhere. In may be desirable to allow importing messages with a leading "From " line for FUSE, even. Additionally, some instances of this regexp needlessly added optional `\r?' (CR) checks ahead of the `\n' (LF) element; but they're pointless anyways since [^\n]* is enough to exclude all non-LF bytes.
2023-09-28lei: don't gzip --rsyncable by default for mbox*
Using and memoizing the usability of `--rsyncable' is unsafe since pigz (or GNU gzip) can be uninstalled and leave a user with a non-rsync-aware gzip implementation in the long-running daemon. So we stop passing --rsyncable by default to pigz/gzip and no longer attempt to check for it (since it was a TOCTTOU error, anyways). Specifying --rsyncable explicitly didn't work, either, and ended up passing `1' to the gzip/pigz argv :x Finally, we now test --rsyncable on the CLI by adding support for it in `lei convert' and testing it in t/lei-convert.t
2021-11-02mbox_reader: do not blindly pass --rsyncable to gzip
FreeBSD gzip does not support --rsyncable, though my VM usually has pigz installed.
2021-09-04lei_to_mail+mbox_reader: fix handling of empty/bogus emails
We may be handling invalid mboxes, so just return no objects in that case. While "lei q" on HTTP(S) externals expects a gzipped mboxrd, there's always a chance something else gzipped can be sent to us. There's also changes to lei_to_mail to better handle emails which lack a body and/or headers (e.g. t/solve/bare.patch) Link: https://public-inbox.org/meta/20210903151500.h72mzcpqixgtytjs@meerkat.local/
2021-03-29lei_input: support compressed mboxes
Since "lei q" and "lei convert" already support writing these compressed inboxes, it makes sense that all mbox readers support them, as well. Using compression is one reliable way to know an mboxrd or mboxo hasn't been unexpectedly truncated.
2021-03-23mbox_reader: add ->reads method to avoid nonsensical formats
Relying on UNIVERSAL::can may cause internal helper methods to be used, which can lead to failures or nonsensical results.
2021-03-16mbox: move mbox_keywords to MboxReader
MboxReader is a more appropriate place for it than LeiStore.
2021-02-12mbox_reader: do not chomp non-blank EOL
It's conceivable some cases won't generate an empty line before an mboxrd or mboxo From_ line. Ensure we can handle that case and don't leave the Eml->{bdy} without a trailing LF character. And drop an unnecessary alarm import while we're in the area.
2021-01-02processpipe: allow synchronous close to set $?
To get rid of the ugly $PublicInbox::DS::in_loop localization in MboxReader, we'll distinguish between ->CLOSE and ->DESTROY with ProcessPipe. If we end up closing via ->DESTROY, we'll assume the caller will want to deal with $? asynchronously via the event loop (or not even care about $?). If we hit ->CLOSE directly, we'll assume the caller called close() and wants to check $? synchronously. Note: wantarray doesn't seem to propagate into tied methods, otherwise I'd be relying on that.
2021-01-01update copyrights for 2021
Using "make update-copyrights" after setting GNULIB_PATH in my config.mak
2021-01-01lei_to_mail: start --augment, dedupe, bz2 and xz
--augment will match the mairix(1) option of the same name to augment existing search results. We'll need to implement deduplication for a better user experience. mutt ships with compressed mbox support for bz2 and xz, at least, so we'll support those out-of-the-box.
2021-01-01mboxreader: new class for reading various mbox formats
This is only lightly-tested against stuff LeiToMail generates and will need real-world tests to validate.