| Date | Commit message (Collapse) |
|---|
|
.dump_csv was added to dump_args for the destructor, but not
initialized properly for the Mwrap.dump API call.
|
|
A weak symbol works fine, here.
|
|
This contains many changes from https://80x24.org/mwrap-perl.git
commit
* Built-in RCU-friendly version of dlmalloc, no more fragile dlsym(3m)
resolution of malloc-family functions in the constructor
* Allocations are now backed by O_TMPFILE on $TMPDIR on modern Linux.
Since mwrap increases memory usage greatly and I needed to use it
on a system where I needed more VM space but lacked the ability
to add swap.
* Configurable C backtrace level via MWRAP=bt:$DEPTH where $DEPTH
is a non-negative integer. Be careful about increasing it, even
a depth of 3-4 can be orders-of-magnitude more expensive in
time and space. This can be changed dynamically at runtime via
local HTTP (see below).
* Embedded per-process local-socket-only HTTP server obsoletes
MwrapRack when combined with mwrap-rproxy from the Perl dist
(set `MWRAP=socket_dir:/dir/of/sockets')
See https://80x24.org/mwrap-perl/20221210015518.272576-4-e@80x24.org/
for more info.
It now supports downloading CSV (suitable for importing into sqlite 3.32.0+)
* License switched to GPL-3+ to be compatible with GNU binutils
since we may take code from addr2line in the future.
* libxxhash supported if XXH3_64bits is available.
|
|
This quiets `undefining the allocator of T_DATA class Mwrap::SourceLocation'
warnings.
|
|
Ruby 3.1 uses mmap, nowadays, and I don't think it's worth the
effort to suport it since mmap and munmap don't require the
symmetry *memalign + free do.
|
|
While I doubt Ruby (nor Perl) would store pathnames longer than
PATH_MAX by default, it's possible `eval' users to specify
whatever path (and line number) they wish to use.
Likely was the case with `# line $FILE' directives in Perl5
which prompted this clamping.
|
|
We can no longer depend on having GVL/GIL when we have a Ruby
execution context, so so make `kbuf' thread-specific to avoid
data corruption.
This was ported from the Perl5 version, since Perl5 has had
MVM/Ractor-like abilities with native threads for decades,
despite the Perl5 ecosystem largely avoids and discourages
threads.
|
|
While FreeBSD 12.3 + Ruby 3.0.4p208 remains broken with mwrap
(likely due to threading bugs in Ruby) at least get the code
more consistently closer to a working state.
Thus we'll remember to account for PTHREAD_MUTEX_INITIALIZER
requiring malloc for HeapPageBody tracking, as we do with other
mutexes.
For reference, the Perl5 port of mwrap seems to have no problems
on FreeBSD 12.3, so it seems down to misbehavior w.r.t. pthreads
usage within Ruby itself. Perl5 on FreeBSD is configured with
threads support, but Perl5 doesn't spawn background threads by
default like Ruby can.
|
|
Most GNU/Linux distros build Ruby with --enable-shared, so
it makes sense to support it properly even if it's not the
default favored by ruby-core.
__attribute__((weak)) on a local function is not weak enough
for the shared library, so we add the extra check for the
function's existence to have_ec_p(), instead.
Tested with ruby 2.7.4p191 (2021-07-07 revision a21a3b7d23) [x86_64-linux-gnu]
on Debian 11 (bullseye) as well as default builds w/o
--enable-shared.
Note: FreeBSD 12.3 appears broken with mwrap and I no longer
have older FreeBSD systems handy.
|
|
Copyright years are unnecessary churn and probably aren't
necessary:
https://www.linuxfoundation.org/blog/copyright-notices-in-open-source-software-projects/
add POP3 and IMAP archive info, and drop the outdated link
to the live demo since I no longer have the resources to run it.
Finally, add rdoc (olddoc) + rsync support
|
|
These warnings are already suppressed in similar ways in the
Perl/XS port.
The only warning which remains is related to cfree, which I'm
leaving alone, for now, due to lack of use/test-cases:
../../../../ext/mwrap/mwrap.c:689:6: warning: ‘cfree’ specifies less restrictive attributes than its target ‘free’: ‘leaf’, ‘nothrow’ [-Wmissing-attributes]
689 | void cfree(void *) __attribute__((alias("free")));
| ^~~~~
/usr/include/stdlib.h:565:13: note: ‘cfree’ target declared here
565 | extern void free (void *__ptr) __THROW;
| ^~~~
|
|
HEAP_PAGE_SIZE no longer estimates malloc overhead in Ruby 3.0.x,
but we can now rely on the GC::INTERNAL_CONSTANTS hash to access
the true value.
We also need to account for Ractors in 3.0+, and thus we need to
rely on thread-specific `ruby_current_ec' instead of process-wide
`ruby_current_execution_context_ptr' (which no longer exists in
3.0+).
Finally, the VM seems prone to making some small immortal
allocations in a few places we were not expecting in 2.7 and
earlier. Account for that and loosen some exact checks to
account for it.
Tested on Ruby 3.0.3 and 3.0.4
|
|
It's not modified by that function, so constify it for
ease-of-reading and review.
|
|
urcu v0.11.4+ introduced commit
7ca7fe9c03 (Make temporary variable in _rcu_dereference non-const, 2021-07-29)
which conflicts with our use of _LGPL_SOURCE. In retrospect,
CMM_LOAD_SHARED and CMM_STORE_SHARED seem sufficient for our use
of the `totals' cds_lfht pointer since the constructur should always
fire before any threads are running.
This is fixed in urcu v0.12.4 and v0.13.2 (released 2022-08-18)
but I suspect older versions will live on in enterprise/LTS
distros for a long while.
Link: https://lore.kernel.org/lttng-dev/20220809181927.GA3718@dcvr/
|
|
We must not forget to release RCU read locks even if the
process will probably die, soon, due to ENOMEM.
I noticed this while working on the Perl5/XS port.
Link: https://80x24.org/mwrap-perl/20191102020331.28050-4-e@80x24.org/
|
|
I mainly use Perl5 (again :P), and sometimes tracking down where
malloc calls happen is necessary. I don't know of any malloc
wrapper interface which is aware of Perl source locations.
Valgrind and similar tools can only figure out C source
locations, which isn't very useful when hacking in Perl.
|
|
Mwrap.total_bytes_allocated/total_bytes_freed API needs docs,
and we need to clarify heap page stats are not reset.
|
|
Since there's no hope of atomicity here anyways, use 64-bit
counters.
|
|
Including the last rb_gc_count() call would be useful, and
avoid showing giant SIZE_MAX initializers for acc.min fields.
|
|
Since there's no hope of atomicity here anyways, use 64-bit
counters.
|
|
Including the last rb_gc_count() call would be useful, and
avoid showing giant SIZE_MAX initializers for acc.min fields.
|
|
This is useful for people who do not want to modify existing
Ruby programs.
|
|
Since we learned to treat heap_page_body allocations specially,
we can also keep track of rare memalign allocations done outside
of the Ruby GC this way.
|
|
free-ing and calling posix_memalign again can cause
fragmentation in glibc malloc (at least):
https://sourceware.org/bugzilla/show_bug.cgi?id=14581
Add statistics to track lifetimes and deathtimes (time between
free and resurrection via posix_memalign).
|
|
They can be accessed via new methods:
Mwrap.total_bytes_allocated
Mwrap.total_bytes_freed
And reset (non-atomically) via Mwrap.reset
Requested-by: Sam Saffron
https://80x24.org/mwrap-public/CAAtdryPDZjXo0QWPznKgRnd7+jQPMBFBOQdfM31KWGUVgGPWeg@mail.gmail.com/
|
|
|
|
Tracking memory used for monitoring code itself is noise,
so give users the power to omit it.
|
|
Users will still pay the cost in memory overhead, but this
gets rid of misleading statistics. Tracking actual Ruby
object allocations is better left to Ruby itself.
|
|
There's nothing a user can do about allocations which happen
in our constructor (aside from not using mwrap), so do not
track them.
|
|
glibc will place the (virtual) address at the end, and this allows us to
find addresses for programs which put '[' in their progname (e.g. ssh,
mogilefsd, unicorn, avahi, ...)
|
|
PTHREAD_MUTEX_INITIALIZER is incomplete, in that it does lazy
initialization when pthread_mutex_init is called. So we call
pthread_mutex_init explicitly, as well as calling rcu_read_lock
during the constructor to initialize mutexes used by urcu-bp.
|
|
An "int" length is enough for the source location length, as
it's not going to exceed PATH_MAX by much.
|
|
Knowing the average/max lifespan (in terms of GC.count) of
allocations can be useful.
|
|
It is not safe to use the new Mwrap[] method with Mwrap.clear,
and the rcu_head overhead in src_loc structs was not worth it.
|
|
This can be useful in apps to analyze what's live and
what's not.
|
|
cmake (as run by the Ruby test suite for RubyGems) uses libjson
and libtasn1, which respectively call malloc (via `new') and
free before our constructor can even fire. Apparently, C++
variable initialization may call "new" outside of any functions;
and those run before any functions with the GCC constructor
attribute.
Disclaimer: I don't know C++
|
|
Ruby source locations can be extremely long and use excessive
stack space. Move it off stack for the temporary key lookup
to reduce the likelyhood of stack overflows. No need to use
TSD (which is stored on stack with GCC anyways); as we have
GVL at that point.
|
|
And rework our internal_memalign interface to mimic the
posix_memalign API; since that's what Ruby favors and
avoids touching errno on errors.
|
|
rcu_read_unlock may clobber errno, so it's our responsibility
to preserve it. Fortunately, ENOMEM is the only error which
real_malloc can fail with.
Also, do not set errno for posix_memalign.
|
|
Oops :x
|
|
We shouldn't try to read NULL pointers :x
|
|
We need to hold the RCU read lock to ensure the liveness
of the associated src_loc struct we'll be inserting into.
|
|
This reduces portability concerns about the various names of
memalign-related functions and should let us save some memory by
not requiring the malloc implementation to align-away the data.
|
|
This increases costs even more, but will allow leak finding.
It will be made optional in the future.
|
|
Users may screw up ordering of arguments :x
|
|
This should avoid indirect function call overhead as the
symbol can be resolved at link time instead of at runtime
with a constructor.
|
|
|
|
Most of this code is independent of Ruby. synchronize_rcu in
Mwrap#clear can take a while, and I would not rule out having a
version of this which works with Perl5 :>
|
|
Hopefully this improves readability somewhat.
|
|
|