From: Valentin Schneider <vschneid@redhat.com>
To: linux-rt-users@vger.kernel.org, linux-kernel@vger.kernel.org
Cc: Sebastian Andrzej Siewior <bigeasy@linutronix.de>,
Thomas Gleixner <tglx@linutronix.de>,
Juri Lelli <juri.lelli@redhat.com>,
Clark Williams <williams@redhat.com>,
"Luis Claudio R. Goncalves" <lgoncalv@redhat.com>
Subject: [RT BUG] Stall caused by eventpoll, rwlocks and CFS bandwidth controller
Date: Thu, 12 Oct 2023 17:07:02 +0200 [thread overview]
Message-ID: <xhsmhttqvnall.mognet@vschneid.remote.csb> (raw)
Hi folks,
We've had reports of stalls happening on our v6.0-ish frankenkernels, and while
we haven't been able to come out with a reproducer (yet), I don't see anything
upstream that would prevent them from happening.
The setup involves eventpoll, CFS bandwidth controller and timer
expiry, and the sequence looks as follows (time-ordered):
p_read (on CPUn, CFS with bandwidth controller active)
======
ep_poll_callback()
read_lock_irqsave()
...
try_to_wake_up() <- enqueue causes an update_curr() + sets need_resched
due to having no more runtime
preempt_enable()
preempt_schedule() <- switch out due to p_read being now throttled
p_write
=======
ep_poll()
write_lock_irq() <- blocks due to having active readers (p_read)
ktimers/n
=========
timerfd_tmrproc()
`\
ep_poll_callback()
`\
read_lock_irqsave() <- blocks due to having active writer (p_write)
From this point we have a circular dependency:
p_read -> ktimers/n (to replenish runtime of p_read)
ktimers/n -> p_write (to let ktimers/n acquire the readlock)
p_write -> p_read (to let p_write acquire the writelock)
IIUC reverting
286deb7ec03d ("locking/rwbase: Mitigate indefinite writer starvation")
should unblock this as the ktimers/n thread wouldn't block, but then we're back
to having the indefinite starvation so I wouldn't necessarily call this a win.
Two options I'm seeing:
- Prevent p_read from being preempted when it's doing the wakeups under the
readlock (icky)
- Prevent ktimers / ksoftirqd (*) from running the wakeups that have
ep_poll_callback() as a wait_queue_entry callback. Punting that to e.g. a
kworker /should/ do.
(*) It's not just timerfd, I've also seen it via net::sock_def_readable -
it should be anything that's pollable.
I'm still scratching my head on this, so any suggestions/comments welcome!
Cheers,
Valentin
reply other threads:[~2023-10-12 15:07 UTC|newest]
Thread overview: [no followups] expand[flat|nested] mbox.gz Atom feed
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=xhsmhttqvnall.mognet@vschneid.remote.csb \
--to=vschneid@redhat.com \
--cc=bigeasy@linutronix.de \
--cc=juri.lelli@redhat.com \
--cc=lgoncalv@redhat.com \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-rt-users@vger.kernel.org \
--cc=tglx@linutronix.de \
--cc=williams@redhat.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).