From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.1 (2015-04-28) on dcvr.yhbt.net X-Spam-Level: X-Spam-ASN: X-Spam-Status: No, score=-4.0 required=3.0 tests=ALL_TRUSTED,BAYES_00 shortcircuit=no autolearn=ham autolearn_force=no version=3.4.1 Received: from localhost (dcvr.yhbt.net [127.0.0.1]) by dcvr.yhbt.net (Postfix) with ESMTP id B2E1B1F597; Thu, 26 Jul 2018 02:46:01 +0000 (UTC) Date: Thu, 26 Jul 2018 02:46:01 +0000 From: Eric Wong To: Sam Saffron Cc: ruby-talk@ruby-lang.org, mwrap-public@80x24.org Subject: Re: [ANN] mwrap 2.0.0 mwrap - LD_PRELOAD malloc wrapper for Ruby Message-ID: <20180726024601.ievg7jdr7blgtff6@dcvr> References: <20180720092516.GA23759@dcvr> <20180720093416.GA27657@dcvr> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline In-Reply-To: List-Id: Sam Saffron wrote: > Just to clarify here, I mean 2 single global totals, not a per row > kind of thing. > > On Thu, Jul 26, 2018 at 11:36 AM, Sam Saffron wrote: > > I am using mwrap to debug a little leak at the moment, one feature > > request I do have though is a tally of totals. > > > > It would be nice if it could keep track of total allocated and total > > released. That way if my RSS is bloating I can tell if it is due to > > fragmentation or if it is due to a genuine leak really quick. Something like the patch below? (Barely tested) Since mwrap doesn't track its own memory usage; this might be useful if you have a lot of cold code paths doing allocations, since RSS might not stabilize quickly in that case. Also, if there's a leaker using a malloc wrapper like Ruby's xmalloc (e.g. https://bugs.ruby-lang.org/issues/14929 ) ; mwrap won't make it easy to track down since it can only safely see the one level up the call stack (using GCC's __builtin_return_address with a non-zero level isn't safe) diff --git a/ext/mwrap/mwrap.c b/ext/mwrap/mwrap.c index acc8960..9bb44d0 100644 --- a/ext/mwrap/mwrap.c +++ b/ext/mwrap/mwrap.c @@ -32,6 +32,8 @@ extern size_t __attribute__((weak)) rb_gc_count(void); extern VALUE __attribute__((weak)) rb_cObject; extern VALUE __attribute__((weak)) rb_yield(VALUE); +static size_t total_bytes_inc, total_bytes_dec; + /* true for glibc/dlmalloc/ptmalloc, not sure about jemalloc */ #define ASSUMED_MALLOC_ALIGNMENT (sizeof(void *) * 2) @@ -327,6 +329,8 @@ static struct src_loc *update_stats_rcu_lock(size_t size, uintptr_t caller) if (caa_unlikely(!totals)) return 0; if (locating++) goto out; /* do not recurse into another *alloc */ + uatomic_add(&total_bytes_inc, size); + rcu_read_lock(); if (has_ec_p()) { int line; @@ -390,6 +394,7 @@ void free(void *p) if (l) { size_t age = generation - h->as.live.gen; + uatomic_add(&total_bytes_dec, h->size); uatomic_set(&h->size, 0); uatomic_add(&l->frees, 1); uatomic_add(&l->age_total, age); @@ -710,12 +715,16 @@ static VALUE mwrap_dump(int argc, VALUE * argv, VALUE mod) return Qnil; } +/* The whole operation is not remotely atomic... */ static void *totals_reset(void *ign) { struct cds_lfht *t; struct cds_lfht_iter iter; struct src_loc *l; + uatomic_set(&total_bytes_inc, 0); + uatomic_set(&total_bytes_dec, 0); + rcu_read_lock(); t = rcu_dereference(totals); cds_lfht_for_each_entry(t, &iter, l, hnode) { @@ -1033,6 +1042,16 @@ static VALUE mwrap_quiet(VALUE mod) return rb_ensure(rb_yield, SIZET2NUM(cur), reset_locating, 0); } +static VALUE total_inc(VALUE mod) +{ + return SIZET2NUM(total_bytes_inc); +} + +static VALUE total_dec(VALUE mod) +{ + return SIZET2NUM(total_bytes_dec); +} + /* * Document-module: Mwrap * @@ -1084,6 +1103,8 @@ void Init_mwrap(void) rb_define_singleton_method(mod, "each", mwrap_each, -1); rb_define_singleton_method(mod, "[]", mwrap_aref, 1); rb_define_singleton_method(mod, "quiet", mwrap_quiet, 0); + rb_define_singleton_method(mod, "total_bytes_allocated", total_inc, 0); + rb_define_singleton_method(mod, "total_bytes_freed", total_dec, 0); rb_define_method(cSrcLoc, "each", src_loc_each, 0); rb_define_method(cSrcLoc, "frees", src_loc_frees, 0); rb_define_method(cSrcLoc, "allocations", src_loc_allocations, 0); diff --git a/test/test_mwrap.rb b/test/test_mwrap.rb index 8425c35..d112b4e 100644 --- a/test/test_mwrap.rb +++ b/test/test_mwrap.rb @@ -272,4 +272,15 @@ class TestMwrap < Test::Unit::TestCase res == :foo or abort 'Mwrap.quiet did not return block result' end; end + + def test_total_bytes + assert_separately(+"#{<<~"begin;"}\n#{<<~'end;'}") + begin; + require 'mwrap' + Mwrap.total_bytes_allocated > 0 or abort 'nothing allocated' + Mwrap.total_bytes_freed > 0 or abort 'nothing freed' + Mwrap.total_bytes_allocated > Mwrap.total_bytes_freed or + abort 'freed more than allocated' + end; + end end