about summary refs log tree commit homepage
DateCommit message (Collapse)
2024-02-14www: cgit: support non-standard cgitrc locations
If publicinbox.cgitrc is set in the config file, we'll ensure cgit sees it as CGIT_CONFIG since the configured publicinbox.cgitrc knob may not be the default path the cgit.cgi binary was configured to use. Furthermore, we'll respect CGIT_CONFIG in the environment if publicinbox.cgitrc is unset in the config file at -httpd/-netd startup.
2024-02-13viewvcs: HTML fixes for commits
The "patch is too large to show" text is now broken by an <hr> to prevent it from being confused as part of a commit message (or having somebody intentionally insert that text in a commit message to confuse readers). A missing </pre> is also necessary before the <hr> tag for the related commit search form.
2024-02-13viewvcs: parallelize commit display
Similar to commit cbe2548c91859dfb923548ea85d8531b90d53dc3 (www_coderepo: use OnDestroy to render summary view, 2023-04-09), we can rely on OnDestroy and Qspawn to run dependencies in a structured way and with some extra parallelism for SMP users. Perl (as opposed to POSIX sh) allows us to easily avoid expensive patch generation for large root commits, and also avoid needless `git patch-id' invocations for patches which are too big to show. Avoiding patch-id alone saved nearly 2s from the linux.git root commit[1] with patch generation enabled and brought response times down to ~6s (still slow). Avoiding patch generation for root commits brings it down to a few hundred milliseconds on a public-facing server (nobody wants a 355MB patch rendered as HTML, right?). [1] torvalds/linux.git 1da177e4c3f41524e886b7f1b8a0c1fc7321cac2
2024-02-10www: quiet errors for git-{archive,http-backend}
SIGPIPE (13) can be quite common with unreliable connections and impatient clients, so just ignore them.
2024-02-09view: decode In-Reply-To comments added by some MUAs
Štěpán Němec <stepnem@smrk.net> wrote: > Eric Wong wrote: > > Subject: [PATCH] view: decode In-Reply-To comments added by Gnus > Or just "some MUAs"? Who knows who else... Yeah, I wouldn't be surprised if there were more... ---8<--- Subject: [PATCH] view: decode In-Reply-To comments added by some MUAs Emacs-based MUAs (e.g. Gnus and rmail) can do it, and maybe some others, too. I noticed it in <https://yhbt.net/lore/git/xmqqr0ho9oi9.fsf@gitster.g/> while scanning for something else.
2024-02-08daemon: quiet Email::Address::XS warnings properly
Setting $SIG{__WARN__} at the top-level no longer has any effect since we localize $SIG{__WARN__} when entering ->event_step on a per-listener basis. Fixes: 60d262483a4d (daemon: use per-listener SIG{__WARN__} callbacks, 2022-08-08)
2024-02-06pop3d: support fcntl locks on OpenBSD i386
The packaged Perl on OpenBSD i386 supports 64-bit file offsets but not 64-bit integer support for 'q' and 'Q' with `pack'. Since servers aren't likely to require lock files larger than 2 GB (we'd need an inbox with >2 billion messages), we can workaround the Perl build limitation with explicit padding. File::FcntlLock isn't packaged for OpenBSD <= 7.4 (but should be in future releases), but I can test i386 OpenBSD on an extremely slow VM. Big endian support can be done, too, but I have no idea if there's 32-bit BE users around nowadays...
2024-02-01lei: sort MH inputs sequentially by default
MH sequence numbers can be analogous to IMAP UIDs and NNTP article numbers (or more like IMAP MSNs with clients which pack). In any case, sort then numerically by default to avoid surprising users who treat NNTP spools and mlmmj archives as MH folders. This gives more coherent git history and resulting NNTP/IMAP numbering when round-tripping MH -> v2 -> (NNTP|IMAP) -> MH
2024-02-01scripts/import_*: update usage to include lei tips
These scripts probably don't offer anything useful now that lei has fleshed out read-only MH support and v2 outputs.
2024-02-01scripts/slrnspool2maildir: use MHreader and LeiToMail
This contains gmane-specific header munging to unmunge the things gmane dones to headers. While we're at it, document the generic `lei convert' invocation for users who don't need the gmane-specific header munging.
2024-02-01import: drop redundant `use' statement
We don't need multiple `use PublicInbox::IO' statements to import a subroutine.
2024-02-01lei convert: explicitly allow --sort for inputs
LeiToMail can't sort v2 output, but sorting MH input (and NNTP spool + mlmmj archives) numerically makes sense.
2024-01-31lei_to_mail: improve SIGPIPE handling
I can't reproduce this in t/lei-sigpipe.t with GIANT_INBOX_DIR. In real-world usage, having a large `lei q -f text ...' output piped to a pager and killing the pager prematurely could trigger: non-fatal error from PublicInbox::LeiToMail $?=256 messages in my terminal. This is because $self->{lei} was becoming undefined in the process cleanup process of git_to_mail. So flip the cleanup logic around and unconditionally check for Git::cleanup state to bail out early. With this change, the `non-fatal error ...' message no longer appears when I stop reading results early.
2024-01-30spawn: support some rlimit uses via Inline::C
BSD::Resource isn't packaged for Alpine (as of 3.19), but we also have optional Inline::C support and already rely on calling setrlimit(2) directly from the Inline::C version of pi_fork_exec.
2024-01-30doc/lei-mail-formats: update MH read-only status
I'm not looking forward to dealing with synchronization problems if we end up dealing with writes...
2024-01-30watch: support incremental updates from MH
The good news (compared to lei) is we only have to worry about imports and don't care about the filename nor keywords, so it's immune to .mh_sequences writing inconsistencies across MH implementations and sequence number packing. We still assume the writer will write the mail file with one of: * rename(2) to create the final sequence number filename * a single write(2) if not relying on rename(2) mlmmj and mutt satisfy these requirements. Python's Lib/mailbox.py may, I'm not sure...
2024-01-30syscall: use pure Perl sendmsg/recvmsg on *BSD
While syscall symbols (e.g. SYS_*) have changed on us in FreeBSD during the history of Sys::Syscall and this project and did bite us in some cases; the actual numbers don't get recycled for new syscalls. We're also fortunate that sendmsg and recvmsg syscalls and associated msghdr and cmsg structs predate the BSD forks and are compatible across all the BSDs I've tried. OpenBSD routes Perl `syscall' through libc; while NetBSD + FreeBSD document procedures for maintaining backwards compatibility. It looks like Dragonfly follows FreeBSD, here. Tested on i386 OpenBSD, and amd64 {Free,Net,Open,Dragonfly}BSD This enables *BSD users to use lei, -cindex and future SCM_RIGHTS-only features without needing Inline::C. [1] https://cvsweb.openbsd.org/src/gnu/usr.bin/perl/gen_syscall_emulator.pl [2] https://www.netbsd.org/docs/internals/en/chap-processes.html#syscall_versioning [3] https://wiki.freebsd.org/AddingSyscalls#Backward_compatibily
2024-01-30syscall: update formatting to match our codebase
Sys::Syscall needs separate patches anyways (if it ever gets updated), and having a mix of indentation styles in our codebase gets confusing. We'll also update cfarm-related comments for the current URL.
2024-01-24view: /$INBOX/ links to topics_{new,active}.html
This makes the new endpoints easier-to-find. The navigation is still at the bottom of the page since I figured having it at the top is too cluttered for users on small terminals.
2024-01-24www_topics: simplify date column mapping
We can rely on SQLite to map `MAX(ds)' to `ds' rather than doing it in Perl, reducing the size of our Perl optree at the (smaller) expense of SQLite bytecode.
2024-01-17www: repolist: support globbing in URL
This can make it easier to find deeply-nested repositories on my mirror of git.kernel.org. It's not perfect, since projects like Linux use several completely different basenames (e.g. linux.git vs vfs.git vs net.git), but it can still help find significant matches further up a tree. I don't expect glob characters to conflict with actual git repositories used by reasonable people, but direct (non-glob) hits are still tried first.
2024-01-17config: glob2re: fix over-matching /**/foo
Noticed while adding wildcard support to WwwCoderepo...
2024-01-17config: don't vivify invalid fields for coderepos
We don't need 404s for non-existent coderepos creating fake (and invalid) entries. I noticed this while working on subsequent changes to support globbing in URLs.
2024-01-17examples/unsubscribe-milter@.service: use KillMode=process
This can be a multi-process daemon, but systemd should only kill the top-level one. And also finish a comment about the User having access to the shared private key.
2024-01-17tests: clarify Email::MIME is only for development
We moved to PublicInbox::Eml a while back and have no plans to go back to using Email::MIME, so don't tempt users and packagers to waste disk space on Email::MIME.
2024-01-11lei_to_mail: show supported mbox formats on error
Users may accidentally or unknowingly write `mbox' and not know we support 4 incompatible mbox variants.
2024-01-11lei+net_reader: show NNTP message in more failures
Showing absolutely nothing when hitting a server requiring authentication is a very bad user experience. While we're at it, use Net::Cmd->message in more places where we experience failure, too.
2024-01-11net_reader: fix NNTP credential use
Clearly this was never tested until now, as passwords being retrieved by git-credential got completely ignored and unused. This enables users to connect to NNTP(S) servers requiring a password.
2024-01-10www: use autodie in more coderepo places
This cuts down on code somewhat (before I add more :x)
2024-01-10address: avoid [ undef, undef ] address pairs
For totally bogus things in address fields, we'll fall back to showing the original entry in the name column when using Email::Address::XS. The pure Perl version differs here, but we'll just let them be different when it comes to handling bogus data.
2024-01-10www: linkify inbox addresses in To/Cc headers
This makes it easier to discover contemporary messages crossposted to other groups within the same WWW instance. The internal cache is necessary for giant threads, and the expiry mechanism is necessary to prevent attackers from trivially OOM-ing.
2024-01-10git: lowercase host in host_prefix_url
This will make it more effective for use as a cache key. I'm not entirely happy with this sub being in the Git module since it's used by lei and command-line tools, but that's for another day to deal with...
2024-01-10test_common: key2sub: don't require final ';' in scripts
I noticed this when I wrote a new (but probably unnecessary) *.t test and `make check-run' failed since I omitted the final semi-colon after `done_testing'.
2024-01-10doc: txt2pre: linkify -extindex(1), dedupe -config(5)
I noticed the HTML manpages didn't have -extindex linkification while checking over the docs. While adding it, I also noticed -config(5) had two entries :x
2024-01-10git: workaround occasional -watch error message
I'm not sure how this happens (perl 5.34.1 on FreeBSD 13.2) but it appears the {sock} check can succeed and then go undef and become unable to call ->owner_pid. This happens when libgit2 is in use, so perhaps that's a factor. In any case, the rest of the tests succeed.
2024-01-04lei: MH: support inotify to detect updates
This should help us deal with MH sequence number packing and invalidating mail_sync.sqlite3.
2024-01-02view: always show strict|loose note w/ multi-roots
For thread skeletons with multiple roots, it makes sense to note the strict|loose delineation even when the first message matches the desired Message-ID.
2024-01-02over: re-sort Subject matches for WWW /T/ endpoint
When retrieving loose (Subject) matches for a thread, we wanted the most recent matches in reverse chronological order. However, when displaying the /T/ endpoint generating the thread skeleton, we prefer ascending chronological order to match the flow of the conversation. Reported-by: Askar Safin <safinaskar@gmail.com> Link: https://public-inbox.org/meta/CAPnZJGAqsh8ZhPaCAy5M2NZVNcWrr_Hr94t32VXiyiTXwD9jRQ@mail.gmail.com/
2023-12-30lei: support reading MH for convert+import+index
The MH format is widely-supported and used by various MUAs such as mutt and sylpheed, and a MH-like format is used by mlmmj for archives, as well. Locking implementations for writes are inconsistent, so this commit doesn't support writes, yet. inotify|EVFILT_VNODE watches aren't supported, yet, but that'll have to come since MH allows packing unused integers and renaming files.
2023-12-29pure Perl inotify support
This is a step towards improving the out-of-the-box experience in achieving notifications without XS, extra downloads, and .so loading + runtime mmap overhead. This also fixes loongarch support of all Linux syscalls due to a bad regexp :x All the reachable Linux architectures listed at <https://portal.cfarm.net/machines/list/> should be supported. At the moment, there appears to be no reachable sparc* Linux machines available to cfarm users. Fixes: b0e5093aa3572a86 (syscall: add support for riscv64, 2022-08-11)
2023-12-16lei: use ->child_error API properly
I noticed this bug while developing another feature and tests were getting SIGHUP (since SIGHUP == 1 on most systems).
2023-12-16lei index: support +L: labels
`lei index' should be capable of indexing the the same way `lei import' does, but without the indexing. I only noticed this omission while developing a new feature.
2023-12-16t/pop3d-limit: use v1 inbox to test on ancient git
We don't need v2 features nor scalability to test POP3 stuff.
2023-12-16git: quiet down `rev-parse --git-path' errors
This fixes t/mda.t with git 1.8.5
2023-12-16cindex: --prune needs git 2.6+
Older versions of git lack --batch-all-objects, and 2.6+ is new enough already since v2, lei, etc all depend on it.
2023-12-16searchidx: quiet down old git patchid
CentOS 7.x ships with git 1.8.5, so unless a CentOS 7.x user enables 3rd-party repos[1], they'll be stuck with a version of git without `--stable' (though I'm becoming skeptical of indexing patchids at all). [1] https://public-inbox.org/meta/20210421151308.yz5hzkgm75klunpe@nitro.local/
2023-12-16tests: quiet uninitialized warnings on CentOS 7.x
Test::More distributed with Perl 5.16.3 on CentOS 7.x expects the `$how_many' argument for `skip' and warns when its uninitialized, so quiet that warning down.
2023-12-15doc: RelNotes: more 2.0 updates
But new ideas keep popping into muh brain :x
2023-12-13t/lei-import: relax EIO regexp
musl uses "I/O error" while glibc uses "Input/output error" I wish something like strerrorname_np(3) were portable and built into Perl so we could just match on /EIO/.
2023-12-13gzip_filter: use OO ->zflush dispatch
While it's not in a code path intended WwwCoderepo and RepoAtom, those classes provide their own ->zflush, this can future-proof our code against future subclasses at a minor performance cost.