All the mail mirrored from lore.kernel.org
 help / color / mirror / Atom feed
From: Dave Chinner <david@fromorbit.com>
To: Brian Foster <bfoster@redhat.com>
Cc: linux-xfs@vger.kernel.org
Subject: Re: [BUG] generic/475 recovery failure(s)
Date: Sat, 12 Jun 2021 08:33:32 +1000	[thread overview]
Message-ID: <20210611223332.GS664593@dread.disaster.area> (raw)
In-Reply-To: <YMOzT1goreWVgo8S@bfoster>

On Fri, Jun 11, 2021 at 03:02:39PM -0400, Brian Foster wrote:
> On Thu, Jun 10, 2021 at 11:14:32AM -0400, Brian Foster wrote:
> > Hi all,
> > 
> > I'm seeing what looks like at least one new generic/475 failure on
> > current for-next. (I've seen one related to an attr buffer that seems to
> > be older and harder to reproduce.). The test devices are a couple ~15GB
> > lvm devices formatted with mkfs defaults. I'm still trying to establish
> > reproducibility, but so far a failure seems fairly reliable within ~30
> > iterations.
> > 
> > The first [1] looks like log recovery failure processing an EFI. The
> > second variant [2] looks like it passes log recovery, but then fails the
> > mount in the COW extent cleanup stage due to a refcountbt problem. I've
> > also seen one that looks like the same free space corruption error as
> > [1], but triggered via the COW recovery codepath in [2], so these could
> > very well be related. A snippet of the dmesg output for each failed
> > mount is appended below.
> > 
> ...
> 
> A couple updates..
> 
> First (as noted on irc), the generic/475 failure is not new as I was
> able to produce it on vanilla 5.13.0-rc4. I'm not quite sure how far
> back that one goes, but Dave noted he's seen it on occasion for some
> time.
> 
> The generic/019 failure I'm seeing does appear to be new as I cannot
> reproduce on 5.13.0-rc4. This failure looks more like silent fs
> corruption. I.e., the test or log recovery doesn't explicitly fail, but
> the post-test xfs_repair check detects corruption. Example xfs_repair
> output is appended below (note that 'xfs_repair -n' actually crashes,
> while destructive repair seems to work). Since this reproduces fairly
> reliably on for-next, I bisected it (while also navigating an unmount
> hang that I don't otherwise have data on) down to facd77e4e38b ("xfs:
> CIL work is serialised, not pipelined"). From a quick glance at that I'm
> not quite sure what the problem is there, just that it doesn't occur
> prior to that particular commit.

I suspect that there's an underlying bug in the overlapping CIL
commit record sequencing. This commit will be the first time we are
actually getting overlapping checkpoints that need ordering via the
commit record writes. Hence I suspect what is being seen here is a
subtle ordering bug that has been in that code since it was first
introduced but never exercised until now..

I haven't had any success in reproducing this yet, I'll keep trying
to see if I can get it to trigger so I can look at it in more
detail...

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

  reply	other threads:[~2021-06-11 22:33 UTC|newest]

Thread overview: 12+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2021-06-10 15:14 [BUG] generic/475 recovery failure(s) Brian Foster
2021-06-11 19:02 ` Brian Foster
2021-06-11 22:33   ` Dave Chinner [this message]
     [not found]     ` <YMdMehWQoBJC9l0W@bfoster>
2021-06-14 12:56       ` Brian Foster
2021-06-14 23:41         ` Dave Chinner
2021-06-15  4:39           ` Dave Chinner
2021-06-16  7:05     ` Dave Chinner
2021-06-16 20:33       ` Brian Foster
2021-06-16 21:05         ` Darrick J. Wong
2021-06-16 22:54           ` Dave Chinner
2021-06-17  1:28             ` Darrick J. Wong
2021-06-17 12:52           ` Brian Foster

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20210611223332.GS664593@dread.disaster.area \
    --to=david@fromorbit.com \
    --cc=bfoster@redhat.com \
    --cc=linux-xfs@vger.kernel.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.