From: Zorro Lang <zlang@redhat.com>
To: "Darrick J. Wong" <djwong@kernel.org>
Cc: fsverity@lists.linux.dev, linux-fsdevel@vger.kernel.org,
linux-xfs@vger.kernel.org, fstests@vger.kernel.org
Subject: Re: [PATCHSET v5.6] fstests: fs-verity support for XFS
Date: Sat, 11 May 2024 13:01:46 +0800 [thread overview]
Message-ID: <20240511050146.vc4jr2gagwjwwhdp@dell-per750-06-vm-08.rhts.eng.pek2.redhat.com> (raw)
In-Reply-To: <171444687971.962488.18035230926224414854.stgit@frogsfrogsfrogs>
Hi Darrick,
Due to only half of this patchset got reviewed, so I'd like to wait for your
later version. I won't pick up part of this patchset to merge this time, I
think better to merge it as an integrated patchset.
Thanks,
Zorro
On Mon, Apr 29, 2024 at 08:19:24PM -0700, Darrick J. Wong wrote:
> Hi all,
>
> This patchset adds support for fsverity to XFS. In keeping with
> Andrey's original design, XFS stores all fsverity metadata in the
> extended attribute data. However, I've made a few changes to the code:
> First, it now caches merkle tree blocks directly instead of abusing the
> buffer cache. This reduces lookup overhead quite a bit, at a cost of
> needing a new shrinker for cached merkle tree blocks.
>
> To reduce the ondisk footprint further, I also made the verity
> enablement code detect trailing zeroes whenever fsverity tells us to
> write a buffer, and elide storing the zeroes. To further reduce the
> footprint of sparse files, I also skip writing merkle tree blocks if the
> block contents are entirely hashes of zeroes.
>
> Next, I implemented more of the tooling around verity, such as debugger
> support, as much fsck support as I can manage without knowing the
> internal format of the fsverity information; and added support for
> xfs_scrub to read fsverity files to validate the consistency of the data
> against the merkle tree.
>
> Finally, I add the ability for administrators to turn off fsverity,
> which might help recovering damaged data from an inconsistent file.
>
> From Andrey Albershteyn:
>
> Here's v5 of my patchset of adding fs-verity support to XFS.
>
> This implementation uses extended attributes to store fs-verity
> metadata. The Merkle tree blocks are stored in the remote extended
> attributes. The names are offsets into the tree.
> From Darrick J. Wong:
>
> This v5.3 patchset builds upon v5.2 of Andrey's patchset to implement
> fsverity for XFS.
>
> The biggest thing that I didn't like in the v5 patchset is the abuse of
> the data device's buffer cache to store the incore version of the merkle
> tree blocks. Not only do verity state flags end up in xfs_buf, but the
> double-alloc flag wastes memory and doesn't remain internally consistent
> if the xattrs shift around.
>
> I replaced all of that with a per-inode xarray that indexes incore
> merkle tree blocks. For cache hits, this dramatically reduces the
> amount of work that xfs has to do to feed fsverity. The per-block
> overhead is much lower (8 bytes instead of ~300 for xfs_bufs), and we no
> longer have to entertain layering violations in the buffer cache. I
> also added a per-filesystem shrinker so that reclaim can cull cached
> merkle tree blocks, starting with the leaf tree nodes.
>
> I've also rolled in some changes recommended by the fsverity maintainer,
> fixed some organization and naming problems in the xfs code, fixed a
> collision in the xfs_inode iflags, and improved dead merkle tree cleanup
> per the discussion of the v5 series. At this point I'm happy enough
> with this code to start integrating and testing it in my trees, so it's
> time to send it out a coherent patchset for comments.
>
> For v5.3, I've added bits and pieces of online and offline repair
> support, reduced the size of partially filled merkle tree blocks by
> removing trailing zeroes, changed the xattr hash function to better
> avoid collisions between merkle tree keys, made the fsverity
> invalidation bitmap unnecessary, and made it so that we can save space
> on sparse verity files by not storing merkle tree blocks that hash
> totally zeroed data blocks.
>
> From Andrey Albershteyn:
>
> Here's v5 of my patchset of adding fs-verity support to XFS.
>
> This implementation uses extended attributes to store fs-verity
> metadata. The Merkle tree blocks are stored in the remote extended
> attributes. The names are offsets into the tree.
>
> If you're going to start using this code, I strongly recommend pulling
> from my git trees, which are linked below.
>
> This has been running on the djcloud for months with no problems. Enjoy!
> Comments and questions are, as always, welcome.
>
> --D
>
> kernel git tree:
> https://git.kernel.org/cgit/linux/kernel/git/djwong/xfs-linux.git/log/?h=fsverity
>
> xfsprogs git tree:
> https://git.kernel.org/cgit/linux/kernel/git/djwong/xfsprogs-dev.git/log/?h=fsverity
>
> fstests git tree:
> https://git.kernel.org/cgit/linux/kernel/git/djwong/xfstests-dev.git/log/?h=fsverity
> ---
> Commits in this patchset:
> * common/verity: enable fsverity for XFS
> * xfs/{021,122}: adapt to fsverity xattrs
> * xfs/122: adapt to fsverity
> * xfs: test xfs_scrub detection and correction of corrupt fsverity metadata
> * xfs: test disabling fsverity
> * common/populate: add verity files to populate xfs images
> ---
> common/populate | 24 +++++++++
> common/verity | 39 ++++++++++++++-
> tests/xfs/021 | 3 +
> tests/xfs/122.out | 3 +
> tests/xfs/1880 | 135 ++++++++++++++++++++++++++++++++++++++++++++++++++++
> tests/xfs/1880.out | 37 ++++++++++++++
> tests/xfs/1881 | 111 +++++++++++++++++++++++++++++++++++++++++++
> tests/xfs/1881.out | 28 +++++++++++
> 8 files changed, 378 insertions(+), 2 deletions(-)
> create mode 100755 tests/xfs/1880
> create mode 100644 tests/xfs/1880.out
> create mode 100755 tests/xfs/1881
> create mode 100644 tests/xfs/1881.out
>
next prev parent reply other threads:[~2024-05-11 5:01 UTC|newest]
Thread overview: 23+ messages / expand[flat|nested] mbox.gz Atom feed top
[not found] <20240430031134.GH360919@frogsfrogsfrogs>
2024-04-30 3:19 ` [PATCHSET v5.6] fstests: fs-verity support for XFS Darrick J. Wong
2024-04-30 3:41 ` [PATCH 1/6] common/verity: enable fsverity " Darrick J. Wong
2024-04-30 12:39 ` Andrey Albershteyn
2024-04-30 15:35 ` Darrick J. Wong
2024-04-30 3:41 ` [PATCH 2/6] xfs/{021,122}: adapt to fsverity xattrs Darrick J. Wong
2024-04-30 12:46 ` Andrey Albershteyn
2024-04-30 15:36 ` Darrick J. Wong
2024-04-30 3:41 ` [PATCH 3/6] xfs/122: adapt to fsverity Darrick J. Wong
2024-04-30 12:45 ` Andrey Albershteyn
2024-04-30 15:37 ` Darrick J. Wong
2024-04-30 3:41 ` [PATCH 4/6] xfs: test xfs_scrub detection and correction of corrupt fsverity metadata Darrick J. Wong
2024-04-30 12:29 ` Andrey Albershteyn
2024-04-30 15:43 ` Darrick J. Wong
2024-04-30 3:42 ` [PATCH 5/6] xfs: test disabling fsverity Darrick J. Wong
2024-04-30 12:56 ` Andrey Albershteyn
2024-04-30 13:11 ` Andrey Albershteyn
2024-04-30 15:48 ` Darrick J. Wong
2024-04-30 18:06 ` Andrey Albershteyn
2024-04-30 3:42 ` [PATCH 6/6] common/populate: add verity files to populate xfs images Darrick J. Wong
2024-04-30 13:22 ` Andrey Albershteyn
2024-04-30 15:49 ` Darrick J. Wong
2024-05-11 5:01 ` Zorro Lang [this message]
2024-05-17 15:56 ` [PATCHSET v5.6] fstests: fs-verity support for XFS Darrick J. Wong
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20240511050146.vc4jr2gagwjwwhdp@dell-per750-06-vm-08.rhts.eng.pek2.redhat.com \
--to=zlang@redhat.com \
--cc=djwong@kernel.org \
--cc=fstests@vger.kernel.org \
--cc=fsverity@lists.linux.dev \
--cc=linux-fsdevel@vger.kernel.org \
--cc=linux-xfs@vger.kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).