From: bugzilla-daemon@kernel.org
To: linux-ext4@vger.kernel.org
Subject: [Bug 217965] ext4(?) regression since 6.5.0 on sata hdd
Date: Fri, 17 Nov 2023 15:39:32 +0000 [thread overview]
Message-ID: <bug-217965-13602-fyCk8yWSBF@https.bugzilla.kernel.org/> (raw)
In-Reply-To: <bug-217965-13602@https.bugzilla.kernel.org/>
https://bugzilla.kernel.org/show_bug.cgi?id=217965
--- Comment #36 from Ojaswin Mujoo (ojaswin.mujoo@ibm.com) ---
Hey Eyal,
So the trace data has given me an idea of what's going on. Basically in ext4 we
maintain a list of FS blocks groups (BGs) where each list will have BGs based
on the order of free blocks (BG with 64 free blocks goes in list order 6. 640
free blocks goes in order 9 list etc). In our case, we are trying to allocate
stripe size blocks at a time ie 640 blocks or roughly 2.5 KB and ext4 tries to
look at the order 9 list to find a BG that is fit to satisfy our request.
Unfortunately there seems to be a lot of BGs in the order 9 list (> 1000) but
most of them dont have enough free blocks to satisfy the request so we keep
looping
and trying to call ext4_mb_good_group() on each of them to see if anyone is
good enough. Once we do find a good enough BG, due to striping we actually try
to look for blocks which are specially aligned to stripe size and once we don't
find it we just start looping in the list again from the beginning (!!).
Although I have a good idea now, I'm not able to point my finger at the exact
change in 6.5 that might have caused this. We did change the allocator to some
extent and it might be related to this but we need to dig a bit more deeper to
confirm.
Would it be possible to share the same perf record again but this time I'm
adding a few more probes and removing -g so we can fit more in 5MBlimit and
also the commands for Linux 6.4 so we can compare whats changed:
Linux 6.5+:
Probe adding commands:
sudo perf probe -a "ext4_mb_find_good_group_avg_frag_lists order"
sudo perf probe -a "ext4_mb_find_good_group_avg_frag_lists:18 cr
iter->bb_group"
sudo perf probe -a "ext4_mb_good_group:20 free fragments ac->ac_g_ex.fe_len
ac->ac_2order"
sudo perf probe -a "ext4_mb_scan_aligned:26 i max"
Record command:
perf record -e probe:ext4_mb_find_good_group_avg_frag_lists_L18 -e
probe:ext4_mb_good_group_L20 -e probe:ext4_mb_find_good_group_avg_frag_lists -e
probe:ext4_mb_ scan_aligned_L26 -e ext4:ext4_mballoc_alloc -p <pid> sleep 20
Linux 6.4.x:
Probe adding commands:
sudo perf probe -a "ext4_mb_choose_next_group_cr1:25 i iter->bb_group"
sudo perf probe -a "ext4_mb_good_group:20 free fragments ac->ac_g_ex.fe_len
ac->ac_2order"
sudo perf probe -a "ext4_mb_scan_aligned:26 i max"
Record command:
sudo perf record -e probe:ext4_mb_choose_next_group_cr1_L25 -e
probe:ext4_mb_good_group_L20 -e probe:ext4_mb_scan_aligned_L26 -e
ext4:ext4_mballoc_alloc -p <pid> sleep 20
Thanks again for all your help on this!
Regards,
ojaswin
--
You may reply to this email to add a comment.
You are receiving this mail because:
You are watching the assignee of the bug.
next prev parent reply other threads:[~2023-11-17 15:39 UTC|newest]
Thread overview: 73+ messages / expand[flat|nested] mbox.gz Atom feed top
2023-10-02 8:10 [Bug 217965] New: ext4(?) regression since 6.5.0 on sata hdd bugzilla-daemon
2023-10-02 8:52 ` [Bug 217965] " bugzilla-daemon
2023-10-02 8:56 ` bugzilla-daemon
2023-10-02 8:57 ` bugzilla-daemon
2023-10-02 15:22 ` bugzilla-daemon
2023-10-02 15:55 ` bugzilla-daemon
2023-10-03 6:50 ` bugzilla-daemon
2023-10-03 6:57 ` bugzilla-daemon
2023-10-03 11:04 ` bugzilla-daemon
2023-10-03 13:42 ` bugzilla-daemon
2023-10-03 14:23 ` bugzilla-daemon
2023-10-04 12:44 ` bugzilla-daemon
2023-10-04 17:23 ` bugzilla-daemon
2023-10-11 7:53 ` bugzilla-daemon
2023-10-11 16:07 ` bugzilla-daemon
2023-10-11 22:23 ` bugzilla-daemon
2023-10-17 10:45 ` bugzilla-daemon
2023-10-17 10:52 ` bugzilla-daemon
2023-10-20 10:24 ` bugzilla-daemon
2023-10-21 10:07 ` bugzilla-daemon
2023-11-13 22:46 ` bugzilla-daemon
2023-11-14 5:55 ` bugzilla-daemon
2023-11-14 6:47 ` bugzilla-daemon
2023-11-14 8:22 ` bugzilla-daemon
2023-11-14 10:29 ` bugzilla-daemon
2023-11-15 4:06 ` bugzilla-daemon
2023-11-15 17:27 ` bugzilla-daemon
2023-11-15 22:39 ` bugzilla-daemon
2023-11-16 3:15 ` bugzilla-daemon
2023-11-16 4:05 ` bugzilla-daemon
2023-11-16 4:31 ` bugzilla-daemon
2023-11-16 4:33 ` bugzilla-daemon
2023-11-16 19:33 ` bugzilla-daemon
2023-11-17 0:01 ` bugzilla-daemon
2023-11-17 0:44 ` bugzilla-daemon
2023-11-17 1:39 ` bugzilla-daemon
2023-11-17 1:40 ` bugzilla-daemon
2023-11-17 15:39 ` bugzilla-daemon [this message]
2023-11-17 23:52 ` bugzilla-daemon
2023-11-17 23:53 ` bugzilla-daemon
2023-11-17 23:57 ` bugzilla-daemon
2023-11-18 12:10 ` bugzilla-daemon
2023-11-18 13:17 ` bugzilla-daemon
2023-11-20 6:11 ` bugzilla-daemon
2023-11-20 8:48 ` bugzilla-daemon
2023-11-27 17:31 ` bugzilla-daemon
2023-12-03 7:33 ` bugzilla-daemon
2023-12-15 11:27 ` bugzilla-daemon
2023-12-23 0:56 ` bugzilla-daemon
2023-12-23 1:48 ` bugzilla-daemon
2023-12-23 12:46 ` bugzilla-daemon
2023-12-23 14:22 ` bugzilla-daemon
2023-12-23 15:47 ` bugzilla-daemon
2023-12-29 20:35 ` bugzilla-daemon
2023-12-29 20:51 ` bugzilla-daemon
2023-12-29 21:05 ` bugzilla-daemon
2023-12-29 22:11 ` bugzilla-daemon
2023-12-29 22:13 ` bugzilla-daemon
2023-12-29 23:07 ` bugzilla-daemon
2023-12-29 23:22 ` bugzilla-daemon
2024-01-02 4:44 ` bugzilla-daemon
2024-01-02 6:44 ` bugzilla-daemon
2024-01-04 5:38 ` bugzilla-daemon
2024-01-04 22:20 ` bugzilla-daemon
2024-01-06 2:11 ` bugzilla-daemon
2024-01-06 3:18 ` bugzilla-daemon
2024-01-07 2:56 ` bugzilla-daemon
2024-02-07 11:44 ` bugzilla-daemon
2024-03-12 20:27 ` bugzilla-daemon
2024-03-17 11:58 ` bugzilla-daemon
2024-03-18 4:45 ` bugzilla-daemon
2024-03-20 6:46 ` bugzilla-daemon
2024-03-23 8:53 ` bugzilla-daemon
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=bug-217965-13602-fyCk8yWSBF@https.bugzilla.kernel.org/ \
--to=bugzilla-daemon@kernel.org \
--cc=linux-ext4@vger.kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.