From: Roman Gushchin <guro@fb.com>
To: Ming Lei <ming.lei@redhat.com>
Cc: Andrew Morton <akpm@linux-foundation.org>,
Tejun Heo <tj@kernel.org>, <linux-fsdevel@vger.kernel.org>,
<linux-kernel@vger.kernel.org>, <linux-mm@kvack.org>,
Alexander Viro <viro@zeniv.linux.org.uk>, Jan Kara <jack@suse.cz>,
Dennis Zhou <dennis@kernel.org>,
Dave Chinner <dchinner@redhat.com>, <cgroups@vger.kernel.org>,
Jan Kara <jack@suse.com>
Subject: Re: [PATCH v9 3/8] writeback, cgroup: increment isw_nr_in_flight before grabbing an inode
Date: Wed, 9 Jun 2021 17:21:14 -0700 [thread overview]
Message-ID: <YMFa+guFw7OFjf3X@carbon.dhcp.thefacebook.com> (raw)
In-Reply-To: <YMA2XEnJrHyVLWrD@T590>
On Wed, Jun 09, 2021 at 11:32:44AM +0800, Ming Lei wrote:
> On Tue, Jun 08, 2021 at 04:02:20PM -0700, Roman Gushchin wrote:
> > isw_nr_in_flight is used do determine whether the inode switch queue
> > should be flushed from the umount path. Currently it's increased
> > after grabbing an inode and even scheduling the switch work. It means
> > the umount path can be walked past cleanup_offline_cgwb() with active
> > inode references, which can result in a "Busy inodes after unmount."
> > message and use-after-free issues (with inode->i_sb which gets freed).
> >
> > Fix it by incrementing isw_nr_in_flight before doing anything with
> > the inode and decrementing in the case when switching wasn't scheduled.
> >
> > The problem hasn't yet been seen in the real life and was discovered
> > by Jan Kara by looking into the code.
> >
> > Suggested-by: Jan Kara <jack@suse.com>
> > Signed-off-by: Roman Gushchin <guro@fb.com>
> > Reviewed-by: Jan Kara <jack@suse.cz>
> > ---
> > fs/fs-writeback.c | 5 +++--
> > 1 file changed, 3 insertions(+), 2 deletions(-)
> >
> > diff --git a/fs/fs-writeback.c b/fs/fs-writeback.c
> > index b6fc13a4962d..4413e005c28c 100644
> > --- a/fs/fs-writeback.c
> > +++ b/fs/fs-writeback.c
> > @@ -505,6 +505,8 @@ static void inode_switch_wbs(struct inode *inode, int new_wb_id)
> > if (!isw)
> > return;
> >
> > + atomic_inc(&isw_nr_in_flight);
>
> smp_mb() may be required for ordering the WRITE in 'atomic_inc(&isw_nr_in_flight)'
> and the following READ on 'inode->i_sb->s_flags & SB_ACTIVE'. Otherwise,
> cgroup_writeback_umount() may observe zero of 'isw_nr_in_flight' because of
> re-order of the two OPs, then miss the flush_workqueue().
>
> Also this barrier should serve as pair of the one added in cgroup_writeback_umount(),
> so maybe this patch should be merged with 2/8.
Hi Ming!
Good point, I agree. How about a patch below?
Thanks!
--
From 282861286074c47907759d80c01419f0d0630dae Mon Sep 17 00:00:00 2001
From: Roman Gushchin <guro@fb.com>
Date: Wed, 9 Jun 2021 14:14:26 -0700
Subject: [PATCH] cgroup, writeback: add smp_mb() to inode_prepare_wbs_switch()
Add a memory barrier between incrementing isw_nr_in_flight
and checking the sb's SB_ACTIVE flag and grabbing an inode in
inode_prepare_wbs_switch(). It's required to prevent grabbing
an inode before incrementing isw_nr_in_flight, otherwise
0 can be obtained as isw_nr_in_flight in cgroup_writeback_umount()
and isw_wq will not be flushed, potentially leading to a memory
corruption.
Added smp_mb() will work in pair with smp_mb() in
cgroup_writeback_umount().
Suggested-by: Ming Lei <ming.lei@redhat.com>
Signed-off-by: Roman Gushchin <guro@fb.com>
---
fs/fs-writeback.c | 8 ++++++++
1 file changed, 8 insertions(+)
diff --git a/fs/fs-writeback.c b/fs/fs-writeback.c
index 545fce68e919..6332b86ca4ed 100644
--- a/fs/fs-writeback.c
+++ b/fs/fs-writeback.c
@@ -513,6 +513,14 @@ static void inode_switch_wbs_work_fn(struct work_struct *work)
static bool inode_prepare_wbs_switch(struct inode *inode,
struct bdi_writeback *new_wb)
{
+ /*
+ * Paired with smp_mb() in cgroup_writeback_umount().
+ * isw_nr_in_flight must be increased before checking SB_ACTIVE and
+ * grabbing an inode, otherwise isw_nr_in_flight can be observed as 0
+ * in cgroup_writeback_umount() and the isw_wq will be not flushed.
+ */
+ smp_mb();
+
/* while holding I_WB_SWITCH, no one else can update the association */
spin_lock(&inode->i_lock);
if (!(inode->i_sb->s_flags & SB_ACTIVE) ||
--
2.31.1
next prev parent reply other threads:[~2021-06-10 0:21 UTC|newest]
Thread overview: 17+ messages / expand[flat|nested] mbox.gz Atom feed top
2021-06-08 23:02 [PATCH v9 0/8] cgroup, blkcg: prevent dirty inodes to pin dying memory cgroups Roman Gushchin
2021-06-08 23:02 ` [PATCH v9 1/8] writeback, cgroup: do not switch inodes with I_WILL_FREE flag Roman Gushchin
2021-06-08 23:02 ` [PATCH v9 2/8] writeback, cgroup: add smp_mb() to cgroup_writeback_umount() Roman Gushchin
2021-06-08 23:02 ` [PATCH v9 3/8] writeback, cgroup: increment isw_nr_in_flight before grabbing an inode Roman Gushchin
2021-06-09 3:32 ` Ming Lei
2021-06-10 0:21 ` Roman Gushchin [this message]
2021-06-10 6:57 ` Ming Lei
2021-06-08 23:02 ` [PATCH v9 4/8] writeback, cgroup: switch to rcu_work API in inode_switch_wbs() Roman Gushchin
2021-06-08 23:02 ` [PATCH v9 5/8] writeback, cgroup: keep list of inodes attached to bdi_writeback Roman Gushchin
2021-06-08 23:02 ` [PATCH v9 6/8] writeback, cgroup: split out the functional part of inode_switch_wbs_work_fn() Roman Gushchin
2021-06-08 23:02 ` [PATCH v9 7/8] writeback, cgroup: support switching multiple inodes at once Roman Gushchin
2021-06-08 23:02 ` [PATCH v9 8/8] writeback, cgroup: release dying cgwbs by switching attached inodes Roman Gushchin
2021-06-09 0:12 ` Andrew Morton
2021-06-09 0:23 ` Roman Gushchin
2021-06-09 0:37 ` Dennis Zhou
2021-06-09 5:34 ` Andrew Morton
2021-06-09 19:53 ` Roman Gushchin
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=YMFa+guFw7OFjf3X@carbon.dhcp.thefacebook.com \
--to=guro@fb.com \
--cc=akpm@linux-foundation.org \
--cc=cgroups@vger.kernel.org \
--cc=dchinner@redhat.com \
--cc=dennis@kernel.org \
--cc=jack@suse.com \
--cc=jack@suse.cz \
--cc=linux-fsdevel@vger.kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-mm@kvack.org \
--cc=ming.lei@redhat.com \
--cc=tj@kernel.org \
--cc=viro@zeniv.linux.org.uk \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).