From: Olivier Langlois <olivier@trillion01.com>
To: Tony Battersby <tonyb@cybernetics.com>,
"Eric W. Biederman" <ebiederm@xmission.com>,
Oleg Nesterov <oleg@redhat.com>
Cc: Linus Torvalds <torvalds@linux-foundation.org>,
Linux Kernel Mailing List <linux-kernel@vger.kernel.org>,
linux-fsdevel <linux-fsdevel@vger.kernel.org>,
io-uring <io-uring@vger.kernel.org>,
Alexander Viro <viro@zeniv.linux.org.uk>,
Jens Axboe <axboe@kernel.dk>,
"Pavel Begunkov>" <asml.silence@gmail.com>
Subject: Re: [PATCH] coredump: Limit what can interrupt coredumps
Date: Wed, 11 Aug 2021 16:47:58 -0400 [thread overview]
Message-ID: <cbfa38bf5536fe37850823c68ea23f6c93dda154.camel@trillion01.com> (raw)
In-Reply-To: <8af373ec-9609-35a4-f185-f9bdc63d39b7@cybernetics.com>
On Tue, 2021-08-10 at 17:48 -0400, Tony Battersby wrote:
> >
> I just ran into this problem also - coredumps from an io_uring
> program
> to a pipe are truncated. But I am using kernel 5.10.57, which does
> NOT
> have commit 12db8b690010 ("entry: Add support for TIF_NOTIFY_SIGNAL")
> or
> commit 06af8679449d ("coredump: Limit what can interrupt
> coredumps").
> Kernel 5.4 works though, so I bisected the problem to commit
> f38c7e3abfba ("io_uring: ensure async buffered read-retry is setup
> properly") in kernel 5.9. Note that my io_uring program uses only
> async
> buffered reads, which may be why this particular commit makes a
> difference to my program.
>
> My io_uring program is a multi-purpose long-running program with many
> threads. Most threads don't use io_uring but a few of them do.
> Normally, my core dumps are piped to a program so that they can be
> compressed before being written to disk, but I can also test writing
> the
> core dumps directly to disk. This is what I have found:
>
> *) Unpatched 5.10.57: if a thread that doesn't use io_uring triggers
> a
> coredump, the core file is written correctly, whether it is written
> to
> disk or piped to a program, even if another thread is using io_uring
> at
> the same time.
>
> *) Unpatched 5.10.57: if a thread that uses io_uring triggers a
> coredump, the core file is truncated, whether written directly to
> disk
> or piped to a program.
>
> *) 5.10.57+backport 06af8679449d: if a thread that uses io_uring
> triggers a coredump, and the core is written directly to disk, then
> it
> is written correctly.
>
> *) 5.10.57+backport 06af8679449d: if a thread that uses io_uring
> triggers a coredump, and the core is piped to a program, then it is
> truncated.
>
> *) 5.10.57+revert f38c7e3abfba: core dumps are written correctly,
> whether written directly to disk or piped to a program.
>
> Tony Battersby
> Cybernetics
>
Tony,
this is super interesting details. I'm leaving for few days so I will
not be able to look into it until I am back but here is my
interpretation of your findings:
f38c7e3abfba makes it more likely that your task ends up in a fd read
wait queue. Previously the io_uring req queuing was failing and
returning EAGAIN. Now it ends up using io uring fast poll.
When the core dump gets written through a pipe, pipe_write must block
waiting for some event. If the task gets waken up by the io_uring wait
queue entry instead, it must somehow make pipe_write fails.
So the problem must be a mix of TIF_NOTIFY_SIGNAL and the fact that
io_uring wait queue entries aren't cleaned up while doing the core
dump.
I have a new modif to try out. I'll hopefully be able to submit a patch
to fix that once I come back (I cannot do it now or else, I'll never
leave ;-))
next prev parent reply other threads:[~2021-08-11 20:48 UTC|newest]
Thread overview: 66+ messages / expand[flat|nested] mbox.gz Atom feed top
[not found] <192c9697e379bf084636a8213108be6c3b948d0b.camel@trillion01.com>
[not found] ` <9692dbb420eef43a9775f425cb8f6f33c9ba2db9.camel@trillion01.com>
[not found] ` <87h7i694ij.fsf_-_@disp2133>
2021-06-09 20:33 ` [RFC] coredump: Do not interrupt dump for TIF_NOTIFY_SIGNAL Linus Torvalds
2021-06-09 20:48 ` Eric W. Biederman
2021-06-09 20:52 ` Linus Torvalds
2021-06-09 21:02 ` Olivier Langlois
2021-06-09 21:05 ` Eric W. Biederman
2021-06-09 21:26 ` Olivier Langlois
2021-06-09 21:56 ` Olivier Langlois
2021-06-10 14:26 ` Eric W. Biederman
2021-06-10 15:17 ` Olivier Langlois
2021-06-10 18:58 ` [CFT}[PATCH] coredump: Limit what can interrupt coredumps Eric W. Biederman
2021-06-10 19:10 ` Linus Torvalds
2021-06-10 19:18 ` Eric W. Biederman
2021-06-10 19:50 ` Linus Torvalds
2021-06-10 20:11 ` [PATCH] " Eric W. Biederman
2021-06-10 21:04 ` Linus Torvalds
2021-06-12 14:36 ` Olivier Langlois
2021-06-12 16:26 ` Jens Axboe
2021-06-14 14:10 ` Oleg Nesterov
2021-06-14 16:37 ` Eric W. Biederman
2021-06-14 16:59 ` Oleg Nesterov
2021-06-15 22:08 ` Eric W. Biederman
2021-06-16 19:23 ` Olivier Langlois
2021-06-16 20:00 ` Eric W. Biederman
2021-06-18 20:05 ` Olivier Langlois
2021-08-05 13:06 ` Olivier Langlois
2021-08-10 21:48 ` Tony Battersby
2021-08-11 20:47 ` Olivier Langlois [this message]
2021-08-12 1:55 ` Jens Axboe
2021-08-12 13:53 ` Tony Battersby
2021-08-15 20:42 ` Olivier Langlois
2021-08-16 13:02 ` Pavel Begunkov
2021-08-16 13:06 ` Pavel Begunkov
2021-08-17 18:15 ` Jens Axboe
2021-08-17 18:24 ` Jens Axboe
2021-08-17 19:29 ` Tony Battersby
2021-08-17 19:59 ` Jens Axboe
2021-08-17 21:28 ` Jens Axboe
2021-08-17 21:39 ` Tony Battersby
2021-08-17 22:05 ` Jens Axboe
2021-08-18 14:37 ` Tony Battersby
2021-08-18 14:46 ` Jens Axboe
2021-08-18 2:57 ` Jens Axboe
2021-08-18 2:58 ` Jens Axboe
2021-08-21 10:08 ` Olivier Langlois
2021-08-21 16:47 ` Olivier Langlois
2021-08-21 16:51 ` Jens Axboe
2021-08-21 17:21 ` Olivier Langlois
2021-08-21 9:52 ` Olivier Langlois
2021-08-21 9:48 ` Olivier Langlois
2021-10-22 14:13 ` [RFC] coredump: Do not interrupt dump for TIF_NOTIFY_SIGNAL Pavel Begunkov
2021-12-24 1:34 ` Olivier Langlois
2021-12-24 10:37 ` Pavel Begunkov
2021-12-24 19:52 ` Eric W. Biederman
2021-12-28 11:24 ` Pavel Begunkov
2022-03-14 23:58 ` Eric W. Biederman
[not found] ` <8218f1a245d054c940e25142fd00a5f17238d078.camel@trillion01.com>
2022-06-01 3:15 ` Jens Axboe
2022-07-20 16:49 ` [PATCH 0/2] coredump: Allow io_uring using apps to dump to pipes Eric W. Biederman
2022-07-20 16:50 ` [PATCH 1/2] signal: Move stopping for the coredump from do_exit into get_signal Eric W. Biederman
2022-07-20 16:51 ` [PATCH 2/2] coredump: Allow coredumps to pipes to work with io_uring Eric W. Biederman
2022-08-22 21:16 ` Olivier Langlois
2022-08-23 3:35 ` Olivier Langlois
2022-08-23 18:22 ` Eric W. Biederman
2022-08-23 18:27 ` Jens Axboe
2022-08-24 15:11 ` Eric W. Biederman
2022-08-24 15:51 ` Jens Axboe
2022-01-05 19:39 ` [RFC] coredump: Do not interrupt dump for TIF_NOTIFY_SIGNAL Olivier Langlois
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=cbfa38bf5536fe37850823c68ea23f6c93dda154.camel@trillion01.com \
--to=olivier@trillion01.com \
--cc=asml.silence@gmail.com \
--cc=axboe@kernel.dk \
--cc=ebiederm@xmission.com \
--cc=io-uring@vger.kernel.org \
--cc=linux-fsdevel@vger.kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=oleg@redhat.com \
--cc=tonyb@cybernetics.com \
--cc=torvalds@linux-foundation.org \
--cc=viro@zeniv.linux.org.uk \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.