From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on dcvr.yhbt.net X-Spam-Level: X-Spam-ASN: X-Spam-Status: No, score=-4.2 required=3.0 tests=ALL_TRUSTED,BAYES_00, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF shortcircuit=no autolearn=ham autolearn_force=no version=3.4.2 Received: from localhost (dcvr.yhbt.net [127.0.0.1]) by dcvr.yhbt.net (Postfix) with ESMTP id 9EA451F93C for ; Wed, 30 Nov 2022 21:08:37 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=80x24.org; s=selector1; t=1669842517; bh=peoqOfF6wss/3vicdJG2ISEk2JF/VXSdQgwY8SsmZ7k=; h=From:To:Subject:Date:In-Reply-To:References:From; b=VHC3r8BM5d0/n3hhrbQxzHCZMoVzwDz+j3q6QlCaIXdi3IZLzYct6Ze2ANTtgPOLS K8UvHxP/1qxS2FrmMoO2a0OLTZfmbs6fIqrxhq5BS3UHcuNLQz/4xN1ePCGZZijtYp 6svaq2LNNwcoR66eL2Xg2YG1fYw/rTDWwSIMcxBM= From: Eric Wong To: mwrap-perl@80x24.org Subject: [PATCH v2 1/2] use file-backed mmap on Linux Date: Wed, 30 Nov 2022 21:08:36 +0000 Message-Id: <20221130210837.17163-2-e@80x24.org> In-Reply-To: <20221130210837.17163-1-e@80x24.org> References: <20221130210837.17163-1-e@80x24.org> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit List-Id: Due to the high memory usage of mwrap, it may be useful to force allocations onto file-backed maps. This means more physical memory for the rest of the system. As opposed to adding swap indiscriminately, this provides the ability to give the equivalent of swap space only to the processes using mwrap. O_TMPFILE in Linux makes supporting this a breeze, since we can't expect to use tmpfile(3) before malloc is initialized. --- mymalloc.h | 47 +++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 47 insertions(+) diff --git a/mymalloc.h b/mymalloc.h index 4904b74..88190d5 100644 --- a/mymalloc.h +++ b/mymalloc.h @@ -24,6 +24,49 @@ #include #include #include +#include +#include +#include +#include + +#if !defined(MWRAP_FILE_BACKED) && defined(__linux__) && defined(O_TMPFILE) +# define MWRAP_FILE_BACKED 1 +#else +# define MWRAP_FILE_BACKED 0 +#endif +#if MWRAP_FILE_BACKED +# include +static void *my_mmap(size_t size) +{ + int flags = MAP_PRIVATE; + const char *tmpdir = getenv("TMPDIR"); + int fd; + void *ret; + + if (!tmpdir) + tmpdir = "/tmp"; + + fd = open(tmpdir, O_TMPFILE|O_RDWR|S_IRUSR); + if (fd < 0) { + flags |= MAP_ANONYMOUS; + } else { + ssize_t w = pwrite(fd, "", 1, size - 1); /* sparse file */ + + if (w <= 0) { /* 0 may be ENOSPC */ + flags |= MAP_ANONYMOUS; + close(fd); + fd = -1; + } + } + ret = mmap(NULL, size, PROT_READ|PROT_WRITE, flags, fd, 0); + if (fd >= 0) { + int err = errno; + close(fd); + errno = err; + } + return ret; +} +#endif /* MWRAP_FILE_BACKED */ /* knobs for dlmalloc */ #define FOOTERS 1 /* required for remote_free_* stuff */ @@ -32,6 +75,10 @@ #define DLMALLOC_EXPORT static inline /* #define NO_MALLOC_STATS 1 */ #define USE_LOCKS 0 /* we do our own global_mtx + ms_tsd */ +#if MWRAP_FILE_BACKED +# define MMAP(size) my_mmap(size) +# define HAVE_MREMAP 0 +#endif #include "dlmalloc_c.h" #undef ABORT /* conflicts with Perl */