From: Johannes Weiner <hannes@cmpxchg.org>
To: Yafang Shao <laoar.shao@gmail.com>
Cc: Yu Zhao <yuzhao@google.com>,
Axel Rasmussen <axelrasmussen@google.com>,
Chris Down <chris@chrisdown.name>,
cgroups@vger.kernel.org, kernel-team@fb.com,
linux-kernel@vger.kernel.org, linux-mm@kvack.org
Subject: Re: MGLRU premature memcg OOM on slow writes
Date: Fri, 15 Mar 2024 10:27:11 -0400 [thread overview]
Message-ID: <20240315142711.GA1944@cmpxchg.org> (raw)
In-Reply-To: <CALOAHbDn0Dbxfhd72d=7-Z_9PjpP_1pXSm3r9daG_XC_f7vFDQ@mail.gmail.com>
On Fri, Mar 15, 2024 at 10:38:31AM +0800, Yafang Shao wrote:
> On Fri, Mar 15, 2024 at 6:23 AM Yu Zhao <yuzhao@google.com> wrote:
> > I'm surprised to see there was 0 pages under writeback:
> > [Wed Mar 13 11:16:48 2024] total_writeback 0
> > What's your dirty limit?
>
> The background dirty threshold is 2G, and the dirty threshold is 4G.
>
> sysctl -w vm.dirty_background_bytes=$((1024 * 1024 * 1024 * 2))
> sysctl -w vm.dirty_bytes=$((1024 * 1024 * 1024 * 4))
>
> >
> > It's unfortunate that the mainline has no per-memcg dirty limit. (We
> > do at Google.)
>
> Per-memcg dirty limit is a useful feature. We also support it in our
> local kernel, but we didn't enable it for this test case.
> It is unclear why the memcg maintainers insist on rejecting the
> per-memcg dirty limit :(
I don't think that assessment is fair. It's just that nobody has
seriously proposed it (at least not that I remember) since the
cgroup-aware writeback was merged in 2015.
We run millions of machines with different workloads, memory sizes,
and IO devices, and don't feel the need to tune the settings for the
global dirty limits away from the defaults.
Cgroups allot those allowances in proportion to observed writeback
speed and available memory in the container. We set IO rate and memory
limits per container, and it adapts as necessary.
If you have an actual usecase, I'm more than willing to hear you
out. I'm sure that the other maintainers feel the same.
If you're proposing it as a workaround for cgroup1 being
architecturally unable to implement proper writeback cache management,
then it's a more difficult argument. That's one of the big reasons why
cgroup2 exists after all.
> > > As of now, it appears that the most effective solution to address this
> > > issue is to revert the commit 14aa8b2d5c2e. Regarding this commit
> > > 14aa8b2d5c2e, its original intention was to eliminate potential SSD
> > > wearout, although there's no concrete data available on how it might
> > > impact SSD longevity. If the concern about SSD wearout is purely
> > > theoretical, it might be reasonable to consider reverting this commit.
> >
> > The SSD wearout problem was real -- it wasn't really due to
> > wakeup_flusher_threads() itself; rather, the original MGLRU code call
> > the function improperly. It needs to be called under more restricted
> > conditions so that it doesn't cause the SDD wearout problem again.
> > However, IMO, wakeup_flusher_threads() is just another bandaid trying
> > to work around a more fundamental problem. There is no guarantee that
> > the flusher will target the dirty pages in the memcg under reclaim,
> > right?
>
> Right, it is a system-wide fluser.
Is it possible it was woken up just too frequently?
Conventional reclaim wakes it based on actually observed dirty pages
off the LRU. I'm not super familiar with MGLRU, but it looks like it
woke it on every generational bump? That might indeed be too frequent,
and doesn't seem related to the writeback cache state.
We're monitoring write rates quite closely due to wearout concern as
well, especially because we use disk swap too. This is the first time
I'm hearing about reclaim-driven wakeups being a concern. (The direct
writepage calls were a huge problem. But not waking the flushers.)
Frankly, I don't think the issue is fixable without bringing the
wakeup back in some form. Even if you had per-cgroup dirty limits. As
soon as you have non-zero dirty pages, you can produce allocation
patterns that drive reclaim into them before background writeback
kicks in.
If reclaim doesn't wake the flushers and waits for writeback, the
premature OOM margin is the size of the background limit - 1.
Yes, cgroup1 and cgroup2 react differently to seeing pages under
writeback: cgroup1 does wait_on_page_writeback(); cgroup2 samples
batches of pages and throttles at a higher level. But both of them
need the flushers woken, or there is nothing to wait for.
Unless you want to wait for dirty expiration :)
next prev parent reply other threads:[~2024-03-15 14:27 UTC|newest]
Thread overview: 18+ messages / expand[flat|nested] mbox.gz Atom feed top
2024-02-09 2:31 MGLRU premature memcg OOM on slow writes Chris Down
2024-02-29 17:28 ` Chris Down
2024-02-29 23:51 ` Axel Rasmussen
2024-03-01 0:30 ` Chris Down
2024-03-08 19:18 ` Axel Rasmussen
2024-03-08 21:22 ` Johannes Weiner
2024-03-11 9:11 ` Yafang Shao
2024-03-12 16:44 ` Axel Rasmussen
2024-03-12 20:07 ` Yu Zhao
2024-03-12 20:11 ` Yu Zhao
2024-03-13 3:33 ` Yafang Shao
2024-03-14 22:23 ` Yu Zhao
2024-03-15 2:38 ` Yafang Shao
2024-03-15 14:27 ` Johannes Weiner [this message]
2024-03-12 21:08 ` Johannes Weiner
2024-03-13 2:08 ` Yu Zhao
2024-03-13 3:22 ` Johannes Weiner
2024-03-13 10:59 ` Hillf Danton
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20240315142711.GA1944@cmpxchg.org \
--to=hannes@cmpxchg.org \
--cc=axelrasmussen@google.com \
--cc=cgroups@vger.kernel.org \
--cc=chris@chrisdown.name \
--cc=kernel-team@fb.com \
--cc=laoar.shao@gmail.com \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-mm@kvack.org \
--cc=yuzhao@google.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).