| Date | Commit message (Collapse) |
|---|
|
Maybe this fixes some t/mwrap-httpd.t reliability problems I've
noticed on FreeBSD...
|
|
This drastically reduces the number of malloc+free calls in the
/$PID/each/$MIN endpoint for large processes by writing out all
the names into one contiguous buffer and having pointers point
inside of it from the h1_src_loc array. IOW, this means we have
parallel arrays working together, instead of one array and
thousands of discreet strings.
This makes /each/2000 in my application roughly 7x faster than
before and 10x faster than the equivalent Devel::Mwrap::PSGI
Perl code. I wasn't exactly happy with the performance of
the C version before this change, but now it's pretty good :>
|
|
Again...
|
|
I'm terrible at this stuff :<
|
|
The syswrite() needs to be replaced by send(.*MSG_NOSIGNAL),
but I'm ignoring SIGPIPE anyways in case connect() somehow
triggers it. In any case, I'm noticing the local socket
created in Perl (not curl) sometimes fails the first send/write
and/or fails the second send/write syscall. Maybe this can
help diagnose things...
|
|
Plack::App::URLMap (used by plackup --path=)
adjusts SCRIPT_NAME and PATH_INFO, but not REQUEST_URI.
We need to adjust REQUEST_URI since we don't want the
URI-unescaped PATH_INFO.
|
|
Persistent connections is probably not going to be a real use
case, but I may end up supporting them anyways.
|
|
The C-only HTTP server (mwrap_httpd) should be able to run in
non-Perl programs and anything which LD_PRELOADs Mwrap.so. This
HTTP server emulates the existing Devel::Mwrap::PSGI app, but
without any Perl or PSGI runtime dependencies. It only binds to
a per-process UNIX stream socket at $socket_dir/$PID.sock where
$socket_dir is set by the `socket_dir:' directive in the MWRAP
environment variable.
URLs for mwrap_httpd are similar to Devel::Mwrap::PSGI ones,
but prefixed with the $PID to simplify use with the reverse
proxy. Thus, "/each/2000" endpoint becomes "/$PID/each/2000"
The PSGI reverse proxy (Devel::Mwrap::Rproxy + mwrap-rproxy)
is a tiny PSGI application designed to expose the mwrap_httpd
UNIX sockets to web browsers such as `w3m'.
The overall configuration allows mwrap-proxy to provide
access to every mwrapped process via the `/$PID/' prefixed
endpoints.
browser <-> mwrap-rproxy <----> mwrapped #1 (on /$socket_dir/$PID_1.sock)
\---> mwrapped #2 (on /$socket_dir/$PID_2.sock)
\---> mwrapped #3 (on /$socket_dir/$PID_3.sock)
...
The process running mwrap-rproxy and Devel::Mwrap::Rproxy does
not require Mwrap.so to be loaded in them, so they can use any
malloc implementation if tracing rproxy isn't necessary.
|
|
This will make it easier to split out tests to separate files in
the future.
|
|
It'll be used for the C HTTP server.
|
|
This will save memory with large projects where files have many
allocation sites. More importantly, the internal change will
make supporting deeper C backtraces (via backtrace(3)) possible
in the future.
|
|
pthread_mutex_lock needs malloc under FreeBSD :<
|
|
Relying on assert() is difficult since we may be linked
into non-Perl processes via LD_PRELOAD, and Perl headers
will have assert() working on Perl-only symbols which cause
runtime failures during symbol resolution.
Instead of tracking down every potentially assert()-ed symbol
and marking it as weak, it's easier to name our own assert
as `mwrap_assert' and use that everywhere during development.
|
|
URCU actually uses a workqueue that pre-spawns a thread at
creation time, so we're safe as long as we call cds_lfht_new
while signals are blocked.
|
|
Since Linux maps are file-backed, it won't provide an increase
in physical memory. Using larger chunks reduces syscalls and
avoids hitting the Linux system-wide /proc/sys/vm/max_map_count
limit.
It's probably safe for all x86-64 systems these days, and 32-bit
is prone to overflow and not a target of this project (though
this project aims to make other projects suitable for
underpowered 32-bit systems).
|
|
Due to the high memory usage of mwrap, it may be useful to force
allocations onto file-backed maps. This means more physical
memory for the rest of the system. As opposed to adding swap
indiscriminately, this provides the ability to give the
equivalent of swap space only to the processes using mwrap.
O_TMPFILE in Linux makes supporting this a breeze, since we
can't expect to use tmpfile(3) before malloc is initialized.
|
|
While this is copied from the pre-zeroed TSD, it won't always
be the case since the TSD area will be used more extensively.
|
|
This seems to provide a 2% speedup or so in my test case
compared to jhash, but we are only picking 32 of the 64 output
bits from XXH3_64bits.
Using the 32-bit XXH32 variant did not show any measurable
improvement over jhash.
|
|
There's no need to worry about race conditions if operating on
idle arenas, so clean them up immediately rather than cleaning
them up after a pthread_create (which may never come).
We'll also inform active threads about the trim earlier so they
have more cycles to react to the lazy trim request.
|
|
We can also limit the scope of its return value since the
pointers inside the return value do not get freed.
|
|
We no longer need to unlock in this function if it fails,
and asserting rcu_read_ongoing() is a good idea to ensure
cds_lfht_lookup() is used properly.
|
|
alloca(3) is difficult for the compiler and static checkers to
deal with, and we already reserve TSD space for storing path
names.
|
|
backtrace_symbols(3) may fail due to malloc failures, so we
must account for it and return `undef' in that case.
|
|
Terminal space is precious with my font sizes.
|
|
kbuf is always `struct src_loc', not `struct alloc_hdr'.
Fortunately, nobody really needs PATH_MAX size for pathnames, so
this wasn't a problem in practice.
|
|
This will make it easier to keep a consistent hash seed
and/or switch to different hashes in the future.
|
|
While 64 has an obvious meaning to me, I reject Moore's law
and have no idea what cache line sizes modern CPUs use.
|
|
Oops :x Fortunately, most allocations are tracked to a Perl
location and the non-Perl code path is infrequently hit.
|
|
This simplifies portability and can eventually give us more
portability involving malloc use. We can also track allocations
earlier since we no longer need to rely on dlsym.
This malloc implementation (based on dlmalloc) also provides us
with wait-free free(3) calls thanks to wfcqueue, so it makes
call_rcu usage far cheaper.
|
|
There's no need to pay O(n) cost to create a short-lived
hash table and the original regexp was obviously wrong :x
|
|
This may be increased in the future if we add C addr2line support.
|
|
rculfhash, call_rcu, and urcu-bp will spawn background
threads. Ensure we spawn all of these threads while signals
are blocked since the underlying codebase may block signals
and rely on signalfd.
|
|
Don't segfault in case we fail to create rculfhash.
|
|
These aliases aren't necessary for modern glibc, at least,
but they also don't hurt. This quiets an old warning about
cfree.
|
|
This is needed to avoid segfaults in non-Perl processes
such as `cmake'.
Fixes: 3197316f33fdc5d4 "create the `totals' rculfhash as early as possible"
|
|
This allows us to start tracking allocations before Perl calls
our BOOT code, which can be far later if it has to boot other
modules first. As an LD_PRELOAD, the constructor can fire
before Perl's main().
|
|
This is required for it to work reliably in FreeBSD if we hit a
code path which calls backtrace_symbols before BOOT:. This will
allow the next commit to setup `totals' rculfhash earlier in the
constructor, rather than BOOT, allowing allocations to be
tracked as early as possible.
|
|
urcu v0.11.4+ introduced commit
7ca7fe9c03 (Make temporary variable in _rcu_dereference non-const, 2021-07-29)
which conflicts with our use of _LGPL_SOURCE. In retrospect,
CMM_LOAD_SHARED and CMM_STORE_SHARED seem sufficient for our use
of the `totals' cds_lfht pointer since the constructur should always
fire before any threads are running.
This is fixed in urcu v0.12.4 and v0.13.2 (released 2022-08-18)
but I suspect older versions will live on in enterprise/LTS
distros for a long while.
Link: https://lore.kernel.org/lttng-dev/20220809181927.GA3718@dcvr/
|
|
This allows us to avoid some #ifdefs inside function bodies.
This cleanup comes from the Ruby version.
|
|
This is to keep up with the Ruby version.
|
|
__thread is a gcc-ism which clang supports, but _Thread_local is
part of C11 and more likely to be supported by future compilers.
|
|
While I doubt Ruby (nor Perl) would store pathnames longer than
PATH_MAX by default, it's possible `eval' users to specify
whatever path (and line number) they wish to use.
Likely was the case with `# line $FILE' directives in Perl5
which prompted this clamping.
|
|
We can no longer depend on having GVL/GIL when we have a Ruby
execution context, so so make `kbuf' thread-specific to avoid
data corruption.
This was ported from the Perl5 version, since Perl5 has had
MVM/Ractor-like abilities with native threads for decades,
despite the Perl5 ecosystem largely avoids and discourages
threads.
|
|
This release now supports --enable-shared builds of Ruby which
is the common setting for GNU/Linux distros. FreeBSD 12.3
support remains broken, but the code is somewhat revised so it
can be more easily fixed and maintained in the future.
`mwrap --help' and and `mwrap --version' are now supported,
and an RDoc error which only happens during `gem install'
is fixed.
* support --enabled-shared builds of Ruby
* extconf.rb: avoid RDoc errors during gem install
* add --version and --help args
* cleanup some FreeBSD-related workarounds
|
|
While FreeBSD 12.3 + Ruby 3.0.4p208 remains broken with mwrap
(likely due to threading bugs in Ruby) at least get the code
more consistently closer to a working state.
Thus we'll remember to account for PTHREAD_MUTEX_INITIALIZER
requiring malloc for HeapPageBody tracking, as we do with other
mutexes.
For reference, the Perl5 port of mwrap seems to have no problems
on FreeBSD 12.3, so it seems down to misbehavior w.r.t. pthreads
usage within Ruby itself. Perl5 on FreeBSD is configured with
threads support, but Perl5 doesn't spawn background threads by
default like Ruby can.
|
|
These may make things easier for new users, and we'll also with
help text if given no args.
We'll programmatically generate version based on `git describe',
but fallback to a hardcoded version if outside of git. We'll
also start appending `-dirty' to the version string to match
git.git conventions.
|
|
RDoc doesn't seem to like <<'' (blank line) as a heredoc
terminator. That said, RDoc scanning extconf.rb during "gem install"
is unexpected and make no sense to me. Normal `rdoc' invocations
seem to ignore extconf.rb.
|
|
Most GNU/Linux distros build Ruby with --enable-shared, so
it makes sense to support it properly even if it's not the
default favored by ruby-core.
__attribute__((weak)) on a local function is not weak enough
for the shared library, so we add the extra check for the
function's existence to have_ec_p(), instead.
Tested with ruby 2.7.4p191 (2021-07-07 revision a21a3b7d23) [x86_64-linux-gnu]
on Debian 11 (bullseye) as well as default builds w/o
--enable-shared.
Note: FreeBSD 12.3 appears broken with mwrap and I no longer
have older FreeBSD systems handy.
|
|
This release adds Ruby 2.7, 3.0, and 3.1 support.
This also fixes breakage in urcu v0.11.4, v0.12.3, and v0.13.1.
While urcu v0.12.4 and v0.13.2 were released 2022-08-18,
old versions will continue to be in distros for a while
(and urcu v0.11.x won't be updated).
There are also several updates ported from the Perl version.
7 changes since v2.1.0 in 2018:
memalign: perform rcu_read_unlock on ENOMEM
workaround breakage from urcu v0.11.4
constify arg for totals_add_rcu
support Ruby 3.0.x
disable HeapPageBody count test for Ruby 3.1
quiet uninitialized and unused variable warnings
various doc updates
|
|
Copyright years are unnecessary churn and probably aren't
necessary:
https://www.linuxfoundation.org/blog/copyright-notices-in-open-source-software-projects/
add POP3 and IMAP archive info, and drop the outdated link
to the live demo since I no longer have the resources to run it.
Finally, add rdoc (olddoc) + rsync support
|