From: Qi Zheng <zhengqi.arch@bytedance.com>
To: cluster-devel.redhat.com
Subject: [Cluster-devel] [PATCH v3 05/49] mm: shrinker: add infrastructure for dynamically allocating shrinker
Date: Sat, 29 Jul 2023 16:47:57 +0800 [thread overview]
Message-ID: <5e50711c-a616-f95f-d6d2-c69627ac3cf0@bytedance.com> (raw)
In-Reply-To: <ZMOx0y+wdHEATDho@corigine.com>
Hi Simon,
On 2023/7/28 20:17, Simon Horman wrote:
> On Thu, Jul 27, 2023 at 04:04:18PM +0800, Qi Zheng wrote:
>> Currently, the shrinker instances can be divided into the following three
>> types:
>>
>> a) global shrinker instance statically defined in the kernel, such as
>> workingset_shadow_shrinker.
>>
>> b) global shrinker instance statically defined in the kernel modules, such
>> as mmu_shrinker in x86.
>>
>> c) shrinker instance embedded in other structures.
>>
>> For case a, the memory of shrinker instance is never freed. For case b,
>> the memory of shrinker instance will be freed after synchronize_rcu() when
>> the module is unloaded. For case c, the memory of shrinker instance will
>> be freed along with the structure it is embedded in.
>>
>> In preparation for implementing lockless slab shrink, we need to
>> dynamically allocate those shrinker instances in case c, then the memory
>> can be dynamically freed alone by calling kfree_rcu().
>>
>> So this commit adds the following new APIs for dynamically allocating
>> shrinker, and add a private_data field to struct shrinker to record and
>> get the original embedded structure.
>>
>> 1. shrinker_alloc()
>>
>> Used to allocate shrinker instance itself and related memory, it will
>> return a pointer to the shrinker instance on success and NULL on failure.
>>
>> 2. shrinker_register()
>>
>> Used to register the shrinker instance, which is same as the current
>> register_shrinker_prepared().
>>
>> 3. shrinker_free()
>>
>> Used to unregister (if needed) and free the shrinker instance.
>>
>> In order to simplify shrinker-related APIs and make shrinker more
>> independent of other kernel mechanisms, subsequent submissions will use
>> the above API to convert all shrinkers (including case a and b) to
>> dynamically allocated, and then remove all existing APIs.
>>
>> This will also have another advantage mentioned by Dave Chinner:
>>
>> ```
>> The other advantage of this is that it will break all the existing
>> out of tree code and third party modules using the old API and will
>> no longer work with a kernel using lockless slab shrinkers. They
>> need to break (both at the source and binary levels) to stop bad
>> things from happening due to using uncoverted shrinkers in the new
>
> nit: uncoverted -> unconverted
Thanks. Will fix.
>
>> setup.
>> ```
>>
>> Signed-off-by: Qi Zheng <zhengqi.arch@bytedance.com>
>
> ...
>
>> diff --git a/mm/shrinker_debug.c b/mm/shrinker_debug.c
>> index f1becfd45853..506257585408 100644
>> --- a/mm/shrinker_debug.c
>> +++ b/mm/shrinker_debug.c
>> @@ -191,6 +191,20 @@ int shrinker_debugfs_add(struct shrinker *shrinker)
>> return 0;
>> }
>>
>> +int shrinker_debugfs_name_alloc(struct shrinker *shrinker, const char *fmt,
>> + va_list ap)
>> +{
>> + shrinker->name = kvasprintf_const(GFP_KERNEL, fmt, ap);
>> +
>> + return shrinker->name ? 0 : -ENOMEM;
>> +}
>> +
>> +void shrinker_debugfs_name_free(struct shrinker *shrinker)
>> +{
>> + kfree_const(shrinker->name);
>> + shrinker->name = NULL;
>> +}
>> +
>
> These functions have no prototype in this file,
> perhaps internal.h should be included?
The compiler can find these implementations, so I don't think there
is a need to include internal.h here?
Thanks,
Qi
>
>> int shrinker_debugfs_rename(struct shrinker *shrinker, const char *fmt, ...)
>> {
>> struct dentry *entry;
>
> ...
WARNING: multiple messages have this Message-ID (diff)
From: Qi Zheng <zhengqi.arch@bytedance.com>
To: Simon Horman <simon.horman@corigine.com>
Cc: kvm@vger.kernel.org, djwong@kernel.org, roman.gushchin@linux.dev,
david@fromorbit.com, dri-devel@lists.freedesktop.org,
virtualization@lists.linux-foundation.org, linux-mm@kvack.org,
dm-devel@redhat.com, linux-mtd@lists.infradead.org,
cel@kernel.org, x86@kernel.org, steven.price@arm.com,
cluster-devel@redhat.com, xen-devel@lists.xenproject.org,
linux-ext4@vger.kernel.org, paulmck@kernel.org,
linux-arm-msm@vger.kernel.org, linux-nfs@vger.kernel.org,
rcu@vger.kernel.org, linux-bcache@vger.kernel.org,
yujie.liu@intel.com, vbabka@suse.cz, linux-raid@vger.kernel.org,
brauner@kernel.org, tytso@mit.edu, gregkh@linuxfoundation.org,
muchun.song@linux.dev, linux-kernel@vger.kernel.org,
linux-f2fs-devel@lists.sourceforge.net,
linux-xfs@vger.kernel.org, senozhatsky@chromium.org,
netdev@vger.kernel.org, linux-fsdevel@vger.kernel.org,
akpm@linux-foundation.org, linux-erofs@lists.ozlabs.org,
linux-btrfs@vger.kernel.org, tkhai@ya.ru
Subject: Re: [Cluster-devel] [PATCH v3 05/49] mm: shrinker: add infrastructure for dynamically allocating shrinker
Date: Sat, 29 Jul 2023 16:47:57 +0800 [thread overview]
Message-ID: <5e50711c-a616-f95f-d6d2-c69627ac3cf0@bytedance.com> (raw)
Message-ID: <20230729084757.K4lWOI5WI0a7fxfB1Hcd33WR4XQm20LRXhecVed84UM@z> (raw)
In-Reply-To: <ZMOx0y+wdHEATDho@corigine.com>
Hi Simon,
On 2023/7/28 20:17, Simon Horman wrote:
> On Thu, Jul 27, 2023 at 04:04:18PM +0800, Qi Zheng wrote:
>> Currently, the shrinker instances can be divided into the following three
>> types:
>>
>> a) global shrinker instance statically defined in the kernel, such as
>> workingset_shadow_shrinker.
>>
>> b) global shrinker instance statically defined in the kernel modules, such
>> as mmu_shrinker in x86.
>>
>> c) shrinker instance embedded in other structures.
>>
>> For case a, the memory of shrinker instance is never freed. For case b,
>> the memory of shrinker instance will be freed after synchronize_rcu() when
>> the module is unloaded. For case c, the memory of shrinker instance will
>> be freed along with the structure it is embedded in.
>>
>> In preparation for implementing lockless slab shrink, we need to
>> dynamically allocate those shrinker instances in case c, then the memory
>> can be dynamically freed alone by calling kfree_rcu().
>>
>> So this commit adds the following new APIs for dynamically allocating
>> shrinker, and add a private_data field to struct shrinker to record and
>> get the original embedded structure.
>>
>> 1. shrinker_alloc()
>>
>> Used to allocate shrinker instance itself and related memory, it will
>> return a pointer to the shrinker instance on success and NULL on failure.
>>
>> 2. shrinker_register()
>>
>> Used to register the shrinker instance, which is same as the current
>> register_shrinker_prepared().
>>
>> 3. shrinker_free()
>>
>> Used to unregister (if needed) and free the shrinker instance.
>>
>> In order to simplify shrinker-related APIs and make shrinker more
>> independent of other kernel mechanisms, subsequent submissions will use
>> the above API to convert all shrinkers (including case a and b) to
>> dynamically allocated, and then remove all existing APIs.
>>
>> This will also have another advantage mentioned by Dave Chinner:
>>
>> ```
>> The other advantage of this is that it will break all the existing
>> out of tree code and third party modules using the old API and will
>> no longer work with a kernel using lockless slab shrinkers. They
>> need to break (both at the source and binary levels) to stop bad
>> things from happening due to using uncoverted shrinkers in the new
>
> nit: uncoverted -> unconverted
Thanks. Will fix.
>
>> setup.
>> ```
>>
>> Signed-off-by: Qi Zheng <zhengqi.arch@bytedance.com>
>
> ...
>
>> diff --git a/mm/shrinker_debug.c b/mm/shrinker_debug.c
>> index f1becfd45853..506257585408 100644
>> --- a/mm/shrinker_debug.c
>> +++ b/mm/shrinker_debug.c
>> @@ -191,6 +191,20 @@ int shrinker_debugfs_add(struct shrinker *shrinker)
>> return 0;
>> }
>>
>> +int shrinker_debugfs_name_alloc(struct shrinker *shrinker, const char *fmt,
>> + va_list ap)
>> +{
>> + shrinker->name = kvasprintf_const(GFP_KERNEL, fmt, ap);
>> +
>> + return shrinker->name ? 0 : -ENOMEM;
>> +}
>> +
>> +void shrinker_debugfs_name_free(struct shrinker *shrinker)
>> +{
>> + kfree_const(shrinker->name);
>> + shrinker->name = NULL;
>> +}
>> +
>
> These functions have no prototype in this file,
> perhaps internal.h should be included?
The compiler can find these implementations, so I don't think there
is a need to include internal.h here?
Thanks,
Qi
>
>> int shrinker_debugfs_rename(struct shrinker *shrinker, const char *fmt, ...)
>> {
>> struct dentry *entry;
>
> ...
next prev parent reply other threads:[~2023-07-29 8:47 UTC|newest]
Thread overview: 71+ messages / expand[flat|nested] mbox.gz Atom feed top
2023-07-27 8:04 [Cluster-devel] [PATCH v3 00/49] use refcount+RCU method to implement lockless slab shrink Qi Zheng
2023-07-27 8:04 ` [Cluster-devel] [PATCH v3 01/49] binder: fix memory leak in binder_init() Qi Zheng
2023-07-27 8:04 ` [Cluster-devel] [PATCH v3 02/49] mm: move some shrinker-related function declarations to mm/internal.h Qi Zheng
2023-07-27 8:04 ` [Cluster-devel] [PATCH v3 03/49] mm: vmscan: move shrinker-related code into a separate file Qi Zheng
2023-07-27 8:04 ` [Cluster-devel] [PATCH v3 04/49] mm: shrinker: remove redundant shrinker_rwsem in debugfs operations Qi Zheng
2023-07-28 8:13 ` Simon Horman
2023-07-28 8:13 ` Simon Horman
2023-07-28 8:19 ` Qi Zheng
2023-07-28 8:19 ` Qi Zheng
2023-07-27 8:04 ` [Cluster-devel] [PATCH v3 05/49] mm: shrinker: add infrastructure for dynamically allocating shrinker Qi Zheng
2023-07-28 12:17 ` Simon Horman
2023-07-28 12:17 ` Simon Horman
2023-07-29 8:47 ` Qi Zheng [this message]
2023-07-29 8:47 ` Qi Zheng
2023-07-27 8:04 ` [Cluster-devel] [PATCH v3 06/49] kvm: mmu: dynamically allocate the x86-mmu shrinker Qi Zheng
2023-07-27 8:04 ` [Cluster-devel] [PATCH v3 07/49] binder: dynamically allocate the android-binder shrinker Qi Zheng
2023-07-27 8:04 ` [Cluster-devel] [PATCH v3 08/49] drm/ttm: dynamically allocate the drm-ttm_pool shrinker Qi Zheng
2023-07-27 8:04 ` [Cluster-devel] [PATCH v3 09/49] xenbus/backend: dynamically allocate the xen-backend shrinker Qi Zheng
2023-07-27 8:04 ` [Cluster-devel] [PATCH v3 10/49] erofs: dynamically allocate the erofs-shrinker Qi Zheng
2023-07-27 8:04 ` [Cluster-devel] [PATCH v3 11/49] f2fs: dynamically allocate the f2fs-shrinker Qi Zheng
2023-07-27 8:04 ` [Cluster-devel] [PATCH v3 12/49] gfs2: dynamically allocate the gfs2-glock shrinker Qi Zheng
2023-07-27 8:04 ` [Cluster-devel] [PATCH v3 13/49] gfs2: dynamically allocate the gfs2-qd shrinker Qi Zheng
2023-07-27 8:04 ` [Cluster-devel] [PATCH v3 14/49] NFSv4.2: dynamically allocate the nfs-xattr shrinkers Qi Zheng
2023-07-27 8:04 ` [Cluster-devel] [PATCH v3 15/49] nfs: dynamically allocate the nfs-acl shrinker Qi Zheng
2023-07-27 9:06 ` Qi Zheng
2023-07-27 8:04 ` [Cluster-devel] [PATCH v3 16/49] nfsd: dynamically allocate the nfsd-filecache shrinker Qi Zheng
2023-07-27 9:08 ` Qi Zheng
2023-07-27 8:04 ` [Cluster-devel] [PATCH v3 17/49] quota: dynamically allocate the dquota-cache shrinker Qi Zheng
2023-07-27 8:04 ` [Cluster-devel] [PATCH v3 18/49] ubifs: dynamically allocate the ubifs-slab shrinker Qi Zheng
2023-07-27 8:04 ` [Cluster-devel] [PATCH v3 19/49] rcu: dynamically allocate the rcu-lazy shrinker Qi Zheng
2023-07-27 8:04 ` [Cluster-devel] [PATCH v3 20/49] rcu: dynamically allocate the rcu-kfree shrinker Qi Zheng
2023-07-27 8:04 ` [Cluster-devel] [PATCH v3 21/49] mm: thp: dynamically allocate the thp-related shrinkers Qi Zheng
2023-07-27 8:04 ` [Cluster-devel] [PATCH v3 22/49] sunrpc: dynamically allocate the sunrpc_cred shrinker Qi Zheng
2023-07-27 9:10 ` Qi Zheng
2023-07-27 8:04 ` [Cluster-devel] [PATCH v3 23/49] mm: workingset: dynamically allocate the mm-shadow shrinker Qi Zheng
2023-07-27 8:04 ` [Cluster-devel] [PATCH v3 24/49] drm/i915: dynamically allocate the i915_gem_mm shrinker Qi Zheng
2023-07-27 8:04 ` [Cluster-devel] [PATCH v3 25/49] drm/msm: dynamically allocate the drm-msm_gem shrinker Qi Zheng
2023-07-27 8:04 ` [Cluster-devel] [PATCH v3 26/49] drm/panfrost: dynamically allocate the drm-panfrost shrinker Qi Zheng
2023-07-27 8:04 ` [Cluster-devel] [PATCH v3 27/49] dm: dynamically allocate the dm-bufio shrinker Qi Zheng
2023-07-27 9:13 ` Qi Zheng
2023-07-27 8:04 ` [Cluster-devel] [PATCH v3 28/49] dm zoned: dynamically allocate the dm-zoned-meta shrinker Qi Zheng
2023-07-27 8:30 ` Damien Le Moal
2023-07-27 8:55 ` Qi Zheng
2023-07-27 10:20 ` Damien Le Moal
2023-07-27 10:32 ` Qi Zheng
2023-07-27 22:59 ` Dave Chinner
2023-07-27 23:48 ` Damien Le Moal
2023-07-27 8:04 ` [Cluster-devel] [PATCH v3 29/49] md/raid5: dynamically allocate the md-raid5 shrinker Qi Zheng
2023-07-27 9:15 ` Qi Zheng
2023-07-27 8:04 ` [Cluster-devel] [PATCH v3 30/49] bcache: dynamically allocate the md-bcache shrinker Qi Zheng
2023-07-27 8:04 ` [Cluster-devel] [PATCH v3 31/49] vmw_balloon: dynamically allocate the vmw-balloon shrinker Qi Zheng
2023-07-27 8:04 ` [Cluster-devel] [PATCH v3 32/49] virtio_balloon: dynamically allocate the virtio-balloon shrinker Qi Zheng
2023-07-27 8:04 ` [Cluster-devel] [PATCH v3 33/49] mbcache: dynamically allocate the mbcache shrinker Qi Zheng
2023-07-27 8:04 ` [Cluster-devel] [PATCH v3 34/49] ext4: dynamically allocate the ext4-es shrinker Qi Zheng
2023-07-27 9:17 ` Qi Zheng
2023-07-27 8:04 ` [Cluster-devel] [PATCH v3 35/49] jbd2, ext4: dynamically allocate the jbd2-journal shrinker Qi Zheng
2023-07-27 8:04 ` [Cluster-devel] [PATCH v3 36/49] nfsd: dynamically allocate the nfsd-client shrinker Qi Zheng
2023-07-27 8:04 ` [Cluster-devel] [PATCH v3 37/49] nfsd: dynamically allocate the nfsd-reply shrinker Qi Zheng
2023-07-27 8:04 ` [Cluster-devel] [PATCH v3 38/49] xfs: dynamically allocate the xfs-buf shrinker Qi Zheng
2023-07-27 8:04 ` [Cluster-devel] [PATCH v3 39/49] xfs: dynamically allocate the xfs-inodegc shrinker Qi Zheng
2023-07-27 8:04 ` [Cluster-devel] [PATCH v3 40/49] xfs: dynamically allocate the xfs-qm shrinker Qi Zheng
2023-07-27 9:19 ` Qi Zheng
2023-07-27 8:04 ` [Cluster-devel] [PATCH v3 41/49] zsmalloc: dynamically allocate the mm-zspool shrinker Qi Zheng
2023-07-27 8:04 ` [Cluster-devel] [PATCH v3 42/49] fs: super: dynamically allocate the s_shrink Qi Zheng
2023-07-27 8:04 ` [Cluster-devel] [PATCH v3 43/49] mm: shrinker: remove old APIs Qi Zheng
2023-07-27 8:04 ` [Cluster-devel] [PATCH v3 44/49] drm/ttm: introduce pool_shrink_rwsem Qi Zheng
2023-07-27 8:04 ` [Cluster-devel] [PATCH v3 45/49] mm: shrinker: add a secondary array for shrinker_info::{map, nr_deferred} Qi Zheng
2023-07-27 8:04 ` [Cluster-devel] [PATCH v3 46/49] mm: shrinker: make global slab shrink lockless Qi Zheng
2023-07-27 8:05 ` [Cluster-devel] [PATCH v3 47/49] mm: shrinker: make memcg " Qi Zheng
2023-07-27 8:05 ` [Cluster-devel] [PATCH v3 48/49] mm: shrinker: hold write lock to reparent shrinker nr_deferred Qi Zheng
2023-07-27 8:05 ` [Cluster-devel] [PATCH v3 49/49] mm: shrinker: convert shrinker_rwsem to mutex Qi Zheng
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=5e50711c-a616-f95f-d6d2-c69627ac3cf0@bytedance.com \
--to=zhengqi.arch@bytedance.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).