| Date | Commit message (Collapse) |
|---|
|
My 32-bit server seems less happy with jemalloc; likely since
munmap is creating holes and it's not using sbrk by default.
jemalloc seems to need large VM space (not actual memory)
to work well, and that isn't a possibility for constrained
32-bit systems.
|
|
The 131072 byte lower bound was the old default before the
sliding mmap window was introduced in modern glibc malloc.
While the sliding mmap window was intended to be faster by
reducing syscalls, zeroing and kernel overhead, it is also prone
to fragmentation from allocation patterns seen in evented Perl
servers.
Individual allocations over 128K are rare in our codebase since
there aren't many messages this large, making any performance
impact tiny. Furthermore, the reduction in fragmentation and
memory use will be a speedup for memory-constrained systems
since they can avoid swap and have more leftover for the page
cache.
|
|
Large string processing + concurrency + caching/memoization
really brings out the worst in glibc malloc :<
|
|
I may be mistaken, but I suspect the reason jemalloc handles
long-lived processes better than glibc is due to granularity
reduction being scaled to larger size classes. This can waste
20% of an individual allocation, but increases the likelyhood
of reuse (without splitting/consolidating into other sizes).
In other words, glibc seems to try too hard to make the best fit
for initial allocations. This ends up being suboptimal over
time as those allocations are freed and similar (but not
identical) allocations come in. jemalloc sacrifices the best
initial fit for better fits over a long process lifetime.
|
|
But new ideas keep popping into muh brain :x
|
|
|
|
-extindex is the most important piece for dealing with many
inboxes, so note it first. Also, frequent use of "git gc" is
important for both loose object performance and reducing memory
mappings.
|
|
git 2.33+ contains important optimizations for the
thousands-of-inboxes case. And combine the Inline::C stuff
with libgit2, since our use of libgit2 requires Inline::C.
|
|
git tends to die when mmap(2) fails on this limit, so let
users know about it. Perhaps git could gracefully fallback.
|
|
v2 onions are insecure, deprecated and going away. v3 names are
unfortunately longer and more difficult to remember, but should
be more resistant to attack than v2 ones.
|
|
Some stuff done, some stuff still needs doing.
|
|
Using "make update-copyrights" after setting GNULIB_PATH in my
config.mak
|
|
And change the documentation reference in -tuning to
point to the -index manpage while we're at it.
|
|
|
|
I've learned a thing or three about btrfs in the past few
weeks and remembered some old HDD things, too.
The Xapian MultiDatabase problem will need to be addressed
for 1.7...
|
|
Determining storage device speed and latencies doesn't
seem portable or even possible with the wide variety
of storage layers in use.
This means we need to write a tuning document and hope
users read and improve on it :P
|