about summary refs log tree commit homepage
path: root/lib/PublicInbox/Daemon.pm
DateCommit message (Collapse)
2020-12-26daemon: support --daemonize without Net::Server::Daemonize
We don't actually need Net::Server::Daemonize to support the --daemonize flag, since the daemonize() sub provided by N::S::D doesn't exactly do the things we want. (cherry picked from commit ab11caeb8ae114087636748df4c956d0e625b237)
2020-09-14sigfd: fix typos and scoping on systems w/o epoll+kqueue
Unfortunately, I'm not sure how easy catching these at compile-time, is. Prototypes do not seem to check these at compile time when crossing packages (not even with exported subroutines).
2020-09-02daemon: support --help/-h in -httpd/imapd/nntpd
For consistency with other commands, though the protocol-specific options should refer users to the manpage.
2020-09-01replace ParentPipe with EOFpipe
ParentPipe was a subset of EOFpipe, except EOFpipe correctly accounts for theoretical(*) spurious wakeups on the pipe. (*) AFAIK, spurious wakeups are/were more likely on TCP sockets due to checksum failures, something that's not a problem on local pipes. We're also not sharing pipes like we do with listen sockets on accept(2), so there's no chance of another process grabbing bytes (unless we have bugs in our code).
2020-08-07syscall: support sparc64 (and maybe other big-endian systems)
Thanks to the GCC compile farm project, we can wire up syscalls for sparc64 and set system-specific SFD_* constants properly. I've FINALLY figured out how to use POSIX::SigSet to generate a usable buffer for the syscall perlfunc. This is required for endian-neutral behavior and relevant to sparc64, at least. There's no need for signalfd-related stuff to be constants, either. signalfd initialization is never a hot path and a stub subroutine for constants uses several KB of memory in the interpreter. We'll drop the needless SEEK_CUR import while we're importing O_NONBLOCK, too.
2020-06-28watch: use signalfd for Maildir watching
We can get rid of the janky wannabe self-using-a-directory-instead-of-pipe thing we needed to workaround Filesys::Notify::Simple being blocking. For existing Maildir users, this should be more robust and immune to missed wakeups for signalfd and kqueue-enabled systems; as well as being immune to BOFHs clearing $TMPDIR and preventing notifications from firing. The IMAP IDLE code still uses normal Perl signals, so it's still vulnerable to missed wakeups. That will be addressed in future commits.
2020-06-13preliminary imap server implementation
It shares a bit of code with NNTP. It's copy+pasted for now since this provides new ground to experiment with APIs for dealing with slow storage and many inboxes.
2020-03-24daemon: unlink .oldbin PID file correctly
We need to track the PID file having ".oldbin" appended to it while a SIGUSR2 upgrade is in progress and ensure it is unlinked on SIGQUIT.
2020-03-24daemon: fix SIGUSR2 upgrade with -W0 (no workers)
Disabling workers via `-W0' blesses the contents of the @listeners array, so we need to ensure we call fcntl on the GLOB ref in ->{sock}. Add tests to ensure USR2 works regardless of whether workers are enabled or not.
2020-03-07daemon: remove unused $parent_pipe variable
We can just create a ParentPipe and let PublicInbox::DS manage its life cycle. While we're at it, favor `\&coderef' over `*coderef' so we're explicit about it being a code ref and not some other ref type.
2020-02-06treewide: run update-copyrights from gnulib for 2019
I didn't wait until September to do it, this year!
2020-01-28daemon: provide TCP_DEFER_ACCEPT for Perl <5.14
Socket::TCP_DEFER_ACCEPT() did not appear in the Socket module distributed with Perl until 5.14, despite it being available since Linux 2.4.
2020-01-08daemon: ignore SIGPIPE while running
This is only needed for IO::Poll users, since users with (signalfd || EVFILT_SIGNAL) support run with SIGPIPE (and all other signals) blocked. Fixes: 81a9a43fb858d197 ("daemon: use sigprocmask to block signals at startup")
2020-01-06treewide: "require" + "use" cleanup and docs
There's a bunch of leftover "require" and "use" statements we no longer need and can get rid of, along with some excessive imports via "use". IO::Handle usage isn't always obvious, so add comments describing why a package loads it. Along the same lines, document the tmpdir support as the reason we depend on File::Temp 0.19, even though every Perl 5.10.1+ user has it. While we're at it, favor "use" over "require", since it it gives us extra compile-time checking.
2019-12-16daemon: drop listeners early in master on graceful shutdown
For users not relying on socket activation via systemd (or similar), we want to drop listeners ASAP so another process can bind to their address. While we're at it, disable TTIN and HUP handlers since we have no chance of starting usable workers without listeners.
2019-12-16daemon: shorten lifetime of listener_names mapping
Keeping a ref to the IO::Socket handle was preventing close(2) from being invoked on graceful shutdown of worker.
2019-12-14daemon: use DESTROY for unlinking --pid-file
This gets rid of the last "END{}" block in our code and cleans up a (temporary) circular reference. Furthermore, ensure the cleanup code still works in all configurations by adding tests and testing both the -W1 (default, 1 worker) and -W0 (no workers) code paths.
2019-12-14ds: move EvCleanup code into DS
EvCleanup only existed since Danga::Socket was a separate component, and cleanup code belongs with the event loop.
2019-11-27httpd|nntpd: avoid missed signal wakeups
Our attempt at using a self-pipe in signal handlers was ineffective, since pure Perl code execution is deferred and Perl doesn't use an internal self-pipe/eventfd. In retrospect, I actually prefer the simplicity of Perl in this regard... We can use sigprocmask() from Perl, so we can introduce signalfd(2) and EVFILT_SIGNAL support on Linux and *BSD-based systems, respectively. These OS primitives allow us to avoid a race where Perl checks for signals right before epoll_wait() or kevent() puts the process to sleep. The (few) systems nowadays without signalfd(2) or IO::KQueue will now see wakeups every second to avoid missed signals.
2019-11-24daemon: avoid race when quitting workers
While the master process has a self-pipe to avoid missing signals, worker processes lack that aside from a pipe to detect master death. That pipe doesn't exist when there's no master process, so it's possible DS::close never finishes because it never woke up from epoll_wait. So create a pipe on the worker_quit signal and force it into epoll/kevent so it wakes up right away.
2019-11-24daemon: use sigprocmask when respawning workers
We need to block signals in workers during respawns until they're ready to receive signals.
2019-11-24daemon: use sigprocmask to block signals at startup
`$SIG{FOO} = "IGNORE"' will cause the daemon to miss signals entirely. Instead, we can use sigprocmask to block signal delivery until we have our signal handlers setup. This closes a race where a PID file can be written for an init script and a signal to be dropped via "IGNORE".
2019-09-09run update-copyrights from gnulib for 2019
2019-07-08daemon: use POSIX and WNOHANG more idiomatically
No point in uglifying our code since we need the POSIX module in many places, anyways.
2019-06-30daemon: warn on inheriting blocking listeners
For users relying on socket activation via service manager (e.g. systemd) and running multiple service instances (@1, @2), we need to ensure configuration of the socket is NonBlocking. Otherwise, service managers such as systemd may clear the O_NONBLOCK flag for a small window where accept/accept4 blocks: public-inbox-nntpd@1 |systemd |public-inbox-nntpd@2 --------------------------+----------------+-------------------- F_SETFL,O_NONBLOCK|O_RDWR | | (not running, yet) |F_SETFL, O_RDWR | |fork+exec @2... | accept(...) # blocks! | |(started by systemd) | |F_SETFL,O_NONBLOCK|O_RDWR | |accept(...) non-blocking It's a very small window where O_NONBLOCK can be cleared, but it exists, and I finally hit it after many years.
2019-06-29parentpipe: make the ->close call more obvious
We can close directly in event_step without bad side effects, and then we also don't need to take a reason arg from worker_quit, since we weren't logging it anywhere.
2019-06-24daemon: use FreeBSD accept filters on non-NNTP
Similar to TCP_DEFER_ACCEPT on Linux, FreeBSD has a 'dataready' accept filter which we can use to reduce wakeups when doing TLS negotiation or plain HTTP. There's also a 'httpready' which we can use for plain HTTP connections.
2019-06-24daemon: set TCP_DEFER_ACCEPT on everything but NNTP
This Linux-specific option can save us some wakeups during the TLS negotiation phase, and it can help with ordinary HTTP, too. Plain NNTP (and in the future, POP3) are the only things which require the server send messages, first.
2019-06-24daemon: use SSL_MODE_RELEASE_BUFFERS
34K per idle connection adds up to large amounts of memory; especially with the speed of malloc nowadays compared to the cost of cache misses or worse, swapping.
2019-06-24daemon: map inherited sockets to well-known schemes
I don't want to specify "--listen" in my systemd .service files, so map 563 to NNTPS automatically (and 443 to HTTPS, but HTTPS support doesn't work, yet).
2019-06-24nntp: NNTPS and NNTP+STARTTLS working
It kinda, barely works, and I'm most happy I got it working without any modifications to the main NNTP::event_step callback thanks to the DS->write(CODE) support we inherited from Danga::Socket.
2019-06-24ds: use and export monotonic now()
All of our internal timing code should use monotonic clocks for consistency against system clock adjustments. This can be shared by our Daemon and NNTP packages.
2019-05-15remove hard Devel::Peek dependency and lazy load for daemons
It's only useful for a corner case in long-running daemons when an admin decides to compact or vacuum a Xapian or SQLite DB. As a result, other scripts should run slightly faster. For instance, this saves about 80ms (2.710s => 2.630s) in t/mda.t on my remote workstation. While we're at it, make sure EvCleanup is properly require'd in Daemon.pm and HTTP.pm and document our use of Devel::Peek.
2019-05-04bundle Danga::Socket and Sys::Syscall
These modules are unmaintained upstream at the moment, but I'll be able to help with the intended maintainer once/if CPAN ownership is transferred. OTOH, we've been waiting for that transfer for several years, now... Changes I intend to make: * EPOLLEXCLUSIVE for Linux * remove unused fields wasting memory * kqueue bugfixes e.g. https://rt.cpan.org/Ticket/Display.html?id=116615 * accept4 support And some lower priority experiments: * switch to EV_ONESHOT / EPOLLONESHOT (incompatible changes) * nginx-style buffering to tmpfile instead of string array * sendfile off tmpfile buffers * io_uring maybe?
2019-02-07daemon: favor Socket.pm in Perl 5.14+ for IPv6
Users on Perl 5.14+ are common, so we can try the bundled Socket (not "Socket6") module before attempting Socket6 for IPv6.
2019-02-07daemon: die properly when Socket6 is missing
We need to keep Unix-socket-only httpd instances working without Socket6. This fixes t/httpd-unix.t with Socket6 uninstalled.
2019-01-10daemon: make Socket6 optional, note about Net::Socket::IP
It looks like Net::Socket::IP comes with Perl 5.20 and later; so we won't have to hassle users with another package to install.
2019-01-05shrink low-bandwidth pipes under Linux
I've hit /proc/sys/fs/pipe-user-pages-* limits on some systems. So stop hogging resources on pipes which don't benefit from giant sizes. Some of these can use eventfd in the future to further reduce resource use.
2018-03-22fix syntax warnings
I keep forgetting to run "make syntax"
2018-03-02evcleanup: disable outside of daemon
We'll be using these in a more OO manner for V2Writable (which doesn't use Danga::Socket), so lets not unnecessarily register cleanup handlers intended for network daemons.
2018-02-07update copyrights for 2018
Using update-copyrights from gnulib While we're at it, use the SPDX identifier for AGPL-3.0+ to ease mechanical processing.
2016-12-12daemon: set $now time for NNTP shutdown
commit 6e238ee3396719e578d6a90e177a71ce9f8c1ca0 ("nntp: respect 3 minute idle time for shutdown") was incomplete, and needed this change to Daemon to be effective. In the future, there will be more common code between NNTP.pm and HTTP.pm
2016-08-02daemon: do not chdir unless daemonizing
As far as most process managers are concerned (e.g. systemd), they should already start in '/'. So avoid making our daemon more complex to run by requiring absolute paths during development.
2016-07-29daemon: re-enable SIGWINCH without setsid
This allows systemd users to use SIGWINCH to temporarily (and gracefully) stop an instance of a service without doing a code reload to bring it back up: # start temporary new service code systemctl start public-inbox-nntpd@2.service # momentarily paralyze original service systemctl kill -s WINCH public-inbox-nntpd@1.service if new_code_at_2_sucks then # restart original workers systemctl kill -s HUP public-inbox-nntpd@1.service else # new is better than old, replace original instance systemctl restart public-inbox-nntpd@1.service fi # cleanup the temporary service systemctl stop public-inbox-nntpd@2.service This partially reverts commit 73d274e83b7d300f31e0cc1ceeacbf73c6c2a1e4 ("daemon: disable SIGWINCH unless explicitly daemonized")
2016-07-06daemon: disable USR2/TTIN/TTOU/WINCH in workers
If using a master/worker setup, a careless user could be trying to signal all processes using "killall". This may trigger bad side-effects; but try to limit the side-effects as much as possible.
2016-06-21daemon: disable SIGWINCH unless explicitly daemonized
Checking stdin/stdout/stderr is not sufficient as the daemon without setsid can still be under the control of a terminal. Unfortunately this means systemd users cannot use SIGWINCH, either.
2016-06-18daemon: be less misleading about graceful shutdown
We do not need to count the httpd.async object against our running client count, that is tied to the socket of the actual client. This prevents misleading sysadmins about connected clients during shutdown.
2016-06-11daemon: reset unused signal handlers to default in child
They're effectively noops anyways, and we don't want to be holding a reference to the read end of the parent pipe.
2016-05-23daemon: ignore SIGWINCH when connected to terminal
Users may change terminal sizes if the process is connected to a terminal, so we can't reasonably expect SIGWINCH to work as intended.
2016-05-21daemon: simplify forking
We shouldn't need sigprocmask unless we're running multiple native threads or using vfork, neither of which is the case, here.