From: "Darrick J. Wong" <djwong@kernel.org>
To: djwong@kernel.org
Cc: linux-xfs@vger.kernel.org
Subject: [PATCH 1/1] xfs: fix potential AGI <-> ILOCK ABBA deadlock in xrep_dinode_findmode_walk_directory
Date: Tue, 26 Mar 2024 18:50:29 -0700 [thread overview]
Message-ID: <171150379387.3216268.6890967813601957901.stgit@frogsfrogsfrogs> (raw)
In-Reply-To: <171150379369.3216268.2525451022899750269.stgit@frogsfrogsfrogs>
From: Darrick J. Wong <djwong@kernel.org>
xfs/399 found the following deadlock when fuzzing core.mode = ones:
/proc/20506/task/20558/stack :
[<0>] xfs_ilock+0xa0/0x240 [xfs]
[<0>] xfs_ilock_data_map_shared+0x1b/0x20 [xfs]
[<0>] xrep_dinode_findmode_walk_directory+0x69/0xe0 [xfs]
[<0>] xrep_dinode_find_mode+0x103/0x2a0 [xfs]
[<0>] xrep_dinode_mode+0x7c/0x120 [xfs]
[<0>] xrep_dinode_core+0xed/0x2b0 [xfs]
[<0>] xrep_dinode_problems+0x10/0x80 [xfs]
[<0>] xrep_inode+0x6c/0xc0 [xfs]
[<0>] xrep_attempt+0x64/0x1d0 [xfs]
[<0>] xfs_scrub_metadata+0x365/0x840 [xfs]
[<0>] xfs_scrubv_metadata+0x282/0x430 [xfs]
[<0>] xfs_ioc_scrubv_metadata+0x149/0x1a0 [xfs]
[<0>] xfs_file_ioctl+0xc68/0x1780 [xfs]
/proc/20506/task/20559/stack :
[<0>] xfs_buf_lock+0x3b/0x110 [xfs]
[<0>] xfs_buf_find_lock+0x66/0x1c0 [xfs]
[<0>] xfs_buf_get_map+0x208/0xc00 [xfs]
[<0>] xfs_buf_read_map+0x5d/0x2c0 [xfs]
[<0>] xfs_trans_read_buf_map+0x1b0/0x4c0 [xfs]
[<0>] xfs_read_agi+0xbd/0x190 [xfs]
[<0>] xfs_ialloc_read_agi+0x47/0x160 [xfs]
[<0>] xfs_imap_lookup+0x69/0x1f0 [xfs]
[<0>] xfs_imap+0x1fc/0x3d0 [xfs]
[<0>] xfs_iget+0x357/0xd50 [xfs]
[<0>] xchk_dir_actor+0x16e/0x330 [xfs]
[<0>] xchk_dir_walk_block+0x164/0x1e0 [xfs]
[<0>] xchk_dir_walk+0x13a/0x190 [xfs]
[<0>] xchk_directory+0x1a2/0x2b0 [xfs]
[<0>] xfs_scrub_metadata+0x2f4/0x840 [xfs]
[<0>] xfs_scrubv_metadata+0x282/0x430 [xfs]
[<0>] xfs_ioc_scrubv_metadata+0x149/0x1a0 [xfs]
[<0>] xfs_file_ioctl+0xc68/0x1780 [xfs]
Thread 20558 holds an AGI buffer and is trying to grab the ILOCK of the
root directory. Thread 20559 holds the root directory ILOCK and is
trying to grab the AGI of an inode that is one of the root directory's
children. The AGI held by 20558 is the same buffer that 20559 is trying
to acquire. In other words, this is an ABBA deadlock.
In general, the lock order is ILOCK and then AGI -- rename does this
while preparing for an operation involving whiteouts or renaming files
out of existence; and unlink does this when moving an inode to the
unlinked list. The only place where we do it in the opposite order is
on the child during an icreate, but at that point the child is marked
INEW and is not visible to other threads.
Work around this deadlock by replacing the blocking ilock attempt with a
nonblocking loop that aborts after 30 seconds. Relax for a jiffy after
a failed lock attempt.
Signed-off-by: Darrick J. Wong <djwong@kernel.org>
---
fs/xfs/scrub/inode_repair.c | 48 ++++++++++++++++++++++++++++++++++++++++++-
1 file changed, 47 insertions(+), 1 deletion(-)
diff --git a/fs/xfs/scrub/inode_repair.c b/fs/xfs/scrub/inode_repair.c
index eab380e95ef40..96c5763dc3839 100644
--- a/fs/xfs/scrub/inode_repair.c
+++ b/fs/xfs/scrub/inode_repair.c
@@ -282,6 +282,50 @@ xrep_dinode_findmode_dirent(
return 0;
}
+/* Try to lock a directory, or wait a jiffy. */
+static inline int
+xrep_dinode_ilock_nowait(
+ struct xfs_inode *dp,
+ unsigned int lock_mode)
+{
+ if (xfs_ilock_nowait(dp, lock_mode))
+ return true;
+
+ schedule_timeout_killable(1);
+ return false;
+}
+
+/*
+ * Try to lock a directory to look for ftype hints. Since we already hold the
+ * AGI buffer, we cannot block waiting for the ILOCK because rename can take
+ * the ILOCK and then try to lock AGIs.
+ */
+STATIC int
+xrep_dinode_trylock_directory(
+ struct xrep_inode *ri,
+ struct xfs_inode *dp,
+ unsigned int *lock_modep)
+{
+ unsigned long deadline = jiffies + msecs_to_jiffies(30000);
+ unsigned int lock_mode;
+ int error = 0;
+
+ do {
+ if (xchk_should_terminate(ri->sc, &error))
+ return error;
+ if (time_is_before_jiffies(deadline))
+ return -EBUSY;
+
+ if (xfs_need_iread_extents(&dp->i_df))
+ lock_mode = XFS_ILOCK_EXCL;
+ else
+ lock_mode = XFS_ILOCK_SHARED;
+ } while (!xrep_dinode_ilock_nowait(dp, lock_mode));
+
+ *lock_modep = lock_mode;
+ return 0;
+}
+
/*
* If this is a directory, walk the dirents looking for any that point to the
* scrub target inode.
@@ -299,7 +343,9 @@ xrep_dinode_findmode_walk_directory(
* Scan the directory to see if there it contains an entry pointing to
* the directory that we are repairing.
*/
- lock_mode = xfs_ilock_data_map_shared(dp);
+ error = xrep_dinode_trylock_directory(ri, dp, &lock_mode);
+ if (error)
+ return error;
/*
* If this directory is known to be sick, we cannot scan it reliably
next prev parent reply other threads:[~2024-03-27 1:50 UTC|newest]
Thread overview: 144+ messages / expand[flat|nested] mbox.gz Atom feed top
2024-03-27 1:40 [PATCHBOMB v30] xfs: online fsck patches for 6.10 Darrick J. Wong
2024-03-27 1:46 ` [PATCHSET 01/15] xfs: bug fixes for 6.9 Darrick J. Wong
2024-03-27 1:50 ` Darrick J. Wong [this message]
2024-03-27 16:56 ` [PATCH 1/1] xfs: fix potential AGI <-> ILOCK ABBA deadlock in xrep_dinode_findmode_walk_directory Christoph Hellwig
2024-03-29 18:38 ` Darrick J. Wong
2024-04-03 5:18 ` [PATCHSET] xfs: bug fixes for 6.9 Darrick J. Wong
2024-04-03 5:18 ` [PATCH 1/3] xfs: pass xfs_buf lookup flags to xfs_*read_agi Darrick J. Wong
2024-04-05 14:53 ` Christoph Hellwig
2024-04-03 5:18 ` [PATCH 2/3] xfs: fix an AGI lock acquisition ordering problem in xrep_dinode_findmode Darrick J. Wong
2024-04-05 14:54 ` Christoph Hellwig
2024-04-05 16:52 ` Christoph Hellwig
2024-04-07 22:34 ` Dave Chinner
2024-04-09 22:51 ` Darrick J. Wong
2024-04-03 5:18 ` [PATCH 3/3] xfs: fix potential AGI <-> ILOCK ABBA deadlock in xrep_dinode_findmode_walk_directory Darrick J. Wong
2024-04-05 3:27 ` [PATCH 4/3] xfs: fix error bailout in xrep_abt_build_new_trees Darrick J. Wong
2024-04-05 5:17 ` Christoph Hellwig
2024-03-27 1:46 ` [PATCHSET v30.1 02/15] xfs: improve log incompat feature handling Darrick J. Wong
2024-03-27 1:50 ` [PATCH 1/2] xfs: only clear log incompat flags at clean unmount Darrick J. Wong
2024-04-07 22:48 ` Dave Chinner
2024-03-27 1:51 ` [PATCH 2/2] xfs: only add log incompat features with explicit permission Darrick J. Wong
2024-04-07 23:00 ` Dave Chinner
2024-04-09 22:53 ` Darrick J. Wong
2024-03-27 1:47 ` [PATCHSET v30.1 03/15] xfs: refactorings for atomic file content exchanges Darrick J. Wong
2024-03-27 1:51 ` [PATCH 1/7] xfs: move inode lease breaking functions to xfs_inode.c Darrick J. Wong
2024-03-27 1:51 ` [PATCH 2/7] xfs: move xfs_iops.c declarations out of xfs_inode.h Darrick J. Wong
2024-03-27 1:51 ` [PATCH 3/7] xfs: declare xfs_file.c symbols in xfs_file.h Darrick J. Wong
2024-03-27 1:52 ` [PATCH 4/7] xfs: create a new helper to return a file's allocation unit Darrick J. Wong
2024-03-27 1:52 ` [PATCH 5/7] xfs: hoist multi-fsb allocation unit detection to a helper Darrick J. Wong
2024-03-27 11:05 ` Christoph Hellwig
2024-04-07 23:07 ` Dave Chinner
2024-04-09 21:09 ` Darrick J. Wong
2024-03-27 1:52 ` [PATCH 6/7] xfs: refactor non-power-of-two alignment checks Darrick J. Wong
2024-03-27 1:52 ` [PATCH 7/7] xfs: constify xfs_bmap_is_written_extent Darrick J. Wong
2024-03-27 1:47 ` [PATCHSET v30.1 04/15] xfs: atomic file content exchanges Darrick J. Wong
2024-03-27 1:53 ` [PATCH 01/15] vfs: export remap and write check helpers Darrick J. Wong
2024-03-27 11:07 ` Christoph Hellwig
2024-03-29 19:45 ` Darrick J. Wong
2024-03-27 1:53 ` [PATCH 02/15] xfs: introduce new file range exchange ioctl Darrick J. Wong
2024-03-27 11:12 ` Christoph Hellwig
2024-03-27 1:53 ` [PATCH 03/15] xfs: create a log incompat flag for atomic file mapping exchanges Darrick J. Wong
2024-04-07 23:17 ` Dave Chinner
2024-04-09 21:12 ` Darrick J. Wong
2024-03-27 1:53 ` [PATCH 04/15] xfs: introduce a file mapping exchange log intent item Darrick J. Wong
2024-04-07 23:51 ` Dave Chinner
2024-04-09 1:18 ` Darrick J. Wong
2024-04-09 3:06 ` Darrick J. Wong
2024-03-27 1:54 ` [PATCH 05/15] xfs: create deferred log items for file mapping exchanges Darrick J. Wong
2024-03-27 1:54 ` [PATCH 06/15] xfs: bind together the front and back ends of the file range exchange code Darrick J. Wong
2024-04-08 0:05 ` Dave Chinner
2024-03-27 1:54 ` [PATCH 07/15] xfs: add error injection to test file mapping exchange recovery Darrick J. Wong
2024-03-27 1:54 ` [PATCH 08/15] xfs: condense extended attributes after a mapping exchange operation Darrick J. Wong
2024-03-27 1:55 ` [PATCH 09/15] xfs: condense directories " Darrick J. Wong
2024-03-27 1:55 ` [PATCH 10/15] xfs: condense symbolic links " Darrick J. Wong
2024-03-27 1:55 ` [PATCH 11/15] xfs: make file range exchange support realtime files Darrick J. Wong
2024-03-27 1:55 ` [PATCH 12/15] xfs: support non-power-of-two rtextsize with exchange-range Darrick J. Wong
2024-03-27 1:56 ` [PATCH 13/15] docs: update swapext -> exchmaps language Darrick J. Wong
2024-03-27 1:56 ` [PATCH 14/15] xfs: introduce new file range commit ioctls Darrick J. Wong
2024-03-27 11:06 ` Christoph Hellwig
2024-03-29 19:45 ` Darrick J. Wong
2024-03-27 1:56 ` [PATCH 15/15] xfs: enable logged file mapping exchange feature Darrick J. Wong
2024-03-27 1:47 ` [PATCHSET v30.1 05/15] xfs: create temporary files for online repair Darrick J. Wong
2024-03-27 1:57 ` [PATCH 1/4] xfs: hide private inodes from bulkstat and handle functions Darrick J. Wong
2024-03-27 11:12 ` Christoph Hellwig
2024-03-27 1:57 ` [PATCH 2/4] xfs: create temporary files and directories for online repair Darrick J. Wong
2024-03-27 1:57 ` [PATCH 3/4] xfs: refactor live buffer invalidation for repairs Darrick J. Wong
2024-03-27 1:57 ` [PATCH 4/4] xfs: add the ability to reap entire inode forks Darrick J. Wong
2024-03-27 1:47 ` [PATCHSET v30.1 06/15] xfs: online repair of realtime summaries Darrick J. Wong
2024-03-27 1:58 ` [PATCH 1/3] xfs: support preallocating and copying content into temporary files Darrick J. Wong
2024-03-27 1:58 ` [PATCH 2/3] xfs: teach the tempfile to set up atomic file content exchanges Darrick J. Wong
2024-03-27 1:58 ` [PATCH 3/3] xfs: online repair of realtime summaries Darrick J. Wong
2024-03-27 1:48 ` [PATCHSET v30.1 07/15] xfs: set and validate dir/attr block owners Darrick J. Wong
2024-03-27 1:58 ` [PATCH 01/10] xfs: add an explicit owner field to xfs_da_args Darrick J. Wong
2024-03-27 1:59 ` [PATCH 02/10] xfs: use the xfs_da_args owner field to set new dir/attr block owner Darrick J. Wong
2024-03-27 1:59 ` [PATCH 03/10] xfs: reduce indenting in xfs_attr_node_list Darrick J. Wong
2024-03-27 11:13 ` Christoph Hellwig
2024-03-28 17:39 ` Darrick J. Wong
2024-03-27 1:59 ` [PATCH 04/10] xfs: validate attr leaf buffer owners Darrick J. Wong
2024-03-27 1:59 ` [PATCH 05/10] xfs: validate attr remote value " Darrick J. Wong
2024-03-27 2:00 ` [PATCH 06/10] xfs: validate dabtree node " Darrick J. Wong
2024-03-27 2:00 ` [PATCH 07/10] xfs: validate directory leaf " Darrick J. Wong
2024-03-27 2:00 ` [PATCH 08/10] xfs: validate explicit directory data " Darrick J. Wong
2024-03-27 2:00 ` [PATCH 09/10] xfs: validate explicit directory block " Darrick J. Wong
2024-03-27 2:01 ` [PATCH 10/10] xfs: validate explicit directory free block owners Darrick J. Wong
2024-03-27 1:48 ` [PATCHSET v30.1 08/15] xfs: online repair of extended attributes Darrick J. Wong
2024-03-27 2:01 ` [PATCH 1/7] xfs: enable discarding of folios backing an xfile Darrick J. Wong
2024-03-27 2:01 ` [PATCH 2/7] xfs: create a blob array data structure Darrick J. Wong
2024-03-27 2:01 ` [PATCH 3/7] xfs: use atomic extent swapping to fix user file fork data Darrick J. Wong
2024-03-27 2:02 ` [PATCH 4/7] xfs: repair extended attributes Darrick J. Wong
2024-03-27 2:02 ` [PATCH 5/7] xfs: scrub should set preen if attr leaf has holes Darrick J. Wong
2024-03-27 2:02 ` [PATCH 6/7] xfs: flag empty xattr leaf blocks for optimization Darrick J. Wong
2024-03-27 2:03 ` [PATCH 7/7] xfs: create an xattr iteration function for scrub Darrick J. Wong
2024-03-27 11:15 ` Christoph Hellwig
2024-03-27 1:48 ` [PATCHSET v30.1 09/15] xfs: online repair of inode unlinked state Darrick J. Wong
2024-03-27 2:03 ` [PATCH 1/2] xfs: ensure unlinked list state is consistent with nlink during scrub Darrick J. Wong
2024-03-27 2:03 ` [PATCH 2/2] xfs: update the unlinked list when repairing link counts Darrick J. Wong
2024-03-27 1:48 ` [PATCHSET v30.1 10/15] xfs: online repair of directories Darrick J. Wong
2024-03-27 2:03 ` [PATCH 1/5] xfs: inactivate directory data blocks Darrick J. Wong
2024-03-27 2:04 ` [PATCH 2/5] xfs: online repair of directories Darrick J. Wong
2024-03-27 2:04 ` [PATCH 3/5] xfs: scan the filesystem to repair a directory dotdot entry Darrick J. Wong
2024-03-27 2:04 ` [PATCH 4/5] xfs: online repair of parent pointers Darrick J. Wong
2024-03-27 2:04 ` [PATCH 5/5] xfs: ask the dentry cache if it knows the parent of a directory Darrick J. Wong
2024-03-27 11:16 ` Christoph Hellwig
2024-03-29 19:52 ` Darrick J. Wong
2024-04-03 5:03 ` [PATCH v30.2 " Darrick J. Wong
2024-04-03 11:43 ` Christoph Hellwig
2024-03-27 1:49 ` [PATCHSET v30.1 11/15] xfs: move orphan files to lost and found Darrick J. Wong
2024-03-27 2:05 ` [PATCH 1/3] xfs: move orphan files to the orphanage Darrick J. Wong
2024-03-27 2:05 ` [PATCH 2/3] xfs: move files to orphanage instead of letting nlinks drop to zero Darrick J. Wong
2024-03-27 2:05 ` [PATCH 3/3] xfs: ensure dentry consistency when the orphanage adopts a file Darrick J. Wong
2024-03-27 1:49 ` [PATCHSET v30.1 12/15] xfs: online repair of symbolic links Darrick J. Wong
2024-03-27 2:05 ` [PATCH 1/1] " Darrick J. Wong
2024-03-27 16:53 ` Christoph Hellwig
2024-03-29 20:44 ` Darrick J. Wong
2024-03-29 20:58 ` Darrick J. Wong
2024-04-03 5:12 ` [PATCHSET v30.2] " Darrick J. Wong
2024-04-03 5:12 ` [PATCH 1/3] xfs: expose xfs_bmap_local_to_extents for online repair Darrick J. Wong
2024-04-03 11:43 ` Christoph Hellwig
2024-04-03 5:12 ` [PATCH 2/3] xfs: pass the owner to xfs_symlink_write_target Darrick J. Wong
2024-04-03 11:43 ` Christoph Hellwig
2024-04-03 5:12 ` [PATCH 3/3] xfs: online repair of symbolic links Darrick J. Wong
2024-04-03 11:44 ` Christoph Hellwig
2024-03-27 1:49 ` [PATCHSET v30.1 13/15] xfs: online fsck of iunlink buckets Darrick J. Wong
2024-03-27 2:06 ` [PATCH 1/3] xfs: check AGI unlinked inode buckets Darrick J. Wong
2024-03-27 2:06 ` [PATCH 2/3] xfs: hoist AGI repair context to a heap object Darrick J. Wong
2024-03-27 2:06 ` [PATCH 3/3] xfs: repair AGI unlinked inode bucket lists Darrick J. Wong
2024-03-27 1:49 ` [PATCHSET v30.1 14/15] xfs: inode-related repair fixes Darrick J. Wong
2024-03-27 2:06 ` [PATCH 1/4] xfs: check unused nlink fields in the ondisk inode Darrick J. Wong
2024-03-27 2:07 ` [PATCH 2/4] xfs: try to avoid allocating from sick inode clusters Darrick J. Wong
2024-03-27 2:07 ` [PATCH 3/4] xfs: pin inodes that would otherwise overflow link count Darrick J. Wong
2024-03-27 2:07 ` [PATCH 4/4] xfs: create subordinate scrub contexts for xchk_metadata_inode_subtype Darrick J. Wong
2024-03-27 1:50 ` [PATCHSET v30.1 15/15] xfs: less heavy locks during fstrim Darrick J. Wong
2024-03-27 2:07 ` [PATCH 1/1] xfs: fix severe performance problems when fstrimming a subset of an AG Darrick J. Wong
2024-03-27 11:35 ` Christoph Hellwig
2024-03-29 21:35 ` Darrick J. Wong
2024-03-30 5:38 ` Christoph Hellwig
2024-03-30 21:15 ` Dave Chinner
2024-03-31 22:44 ` Darrick J. Wong
2024-03-27 22:15 ` Dave Chinner
2024-03-29 22:51 ` Darrick J. Wong
2024-03-30 21:51 ` Dave Chinner
2024-03-31 22:44 ` Darrick J. Wong
2024-04-01 22:12 ` Dave Chinner
2024-04-03 5:07 ` [PATCH v30.2 " Darrick J. Wong
2024-04-04 21:46 ` Dave Chinner
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=171150379387.3216268.6890967813601957901.stgit@frogsfrogsfrogs \
--to=djwong@kernel.org \
--cc=linux-xfs@vger.kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.