From: Hans Holmberg <Hans.Holmberg@wdc.com>
To: "fstests@vger.kernel.org" <fstests@vger.kernel.org>
Cc: "linux-btrfs@vger.kernel.org" <linux-btrfs@vger.kernel.org>,
"Damien Le Moal" <Damien.LeMoal@wdc.com>,
"Matias Bjørling" <Matias.Bjorling@wdc.com>,
"Naohiro Aota" <Naohiro.Aota@wdc.com>,
"Johannes Thumshirn" <Johannes.Thumshirn@wdc.com>,
"hch@lst.de" <hch@lst.de>,
"Hans Holmberg" <Hans.Holmberg@wdc.com>
Subject: [PATCH] generic: add gc stress test
Date: Mon, 15 Apr 2024 11:23:24 +0000 [thread overview]
Message-ID: <20240415112259.21760-1-hans.holmberg@wdc.com> (raw)
This test stresses garbage collection for file systems by first filling
up a scratch mount to a specific usage point with files of random size,
then doing overwrites in parallel with deletes to fragment the backing
storage, forcing reclaim.
Signed-off-by: Hans Holmberg <hans.holmberg@wdc.com>
---
Test results in my setup (kernel 6.8.0-rc4+)
f2fs on zoned nullblk: pass (77s)
f2fs on conventional nvme ssd: pass (13s)
btrfs on zoned nublk: fails (-ENOSPC)
btrfs on conventional nvme ssd: fails (-ENOSPC)
xfs on conventional nvme ssd: pass (8s)
Johannes(cc) is working on the btrfs ENOSPC issue.
tests/generic/744 | 124 ++++++++++++++++++++++++++++++++++++++++++
tests/generic/744.out | 6 ++
2 files changed, 130 insertions(+)
create mode 100755 tests/generic/744
create mode 100644 tests/generic/744.out
diff --git a/tests/generic/744 b/tests/generic/744
new file mode 100755
index 000000000000..2c7ab76bf8b1
--- /dev/null
+++ b/tests/generic/744
@@ -0,0 +1,124 @@
+#! /bin/bash
+# SPDX-License-Identifier: GPL-2.0
+# Copyright (c) 2024 Western Digital Corporation. All Rights Reserved.
+#
+# FS QA Test No. 744
+#
+# Inspired by btrfs/273 and generic/015
+#
+# This test stresses garbage collection in file systems
+# by first filling up a scratch mount to a specific usage point with
+# files of random size, then doing overwrites in parallel with
+# deletes to fragment the backing zones, forcing reclaim.
+
+. ./common/preamble
+_begin_fstest auto
+
+# real QA test starts here
+
+_require_scratch
+
+# This test requires specific data space usage, skip if we have compression
+# enabled.
+_require_no_compress
+
+M=$((1024 * 1024))
+min_fsz=$((1 * ${M}))
+max_fsz=$((256 * ${M}))
+bs=${M}
+fill_percent=95
+overwrite_percentage=20
+seq=0
+
+_create_file() {
+ local file_name=${SCRATCH_MNT}/data_$1
+ local file_sz=$2
+ local dd_extra=$3
+
+ POSIXLY_CORRECT=yes dd if=/dev/zero of=${file_name} \
+ bs=${bs} count=$(( $file_sz / ${bs} )) \
+ status=none $dd_extra 2>&1
+
+ status=$?
+ if [ $status -ne 0 ]; then
+ echo "Failed writing $file_name" >>$seqres.full
+ exit
+ fi
+}
+
+_total_M() {
+ local total=$(stat -f -c '%b' ${SCRATCH_MNT})
+ local bs=$(stat -f -c '%S' ${SCRATCH_MNT})
+ echo $(( ${total} * ${bs} / ${M}))
+}
+
+_used_percent() {
+ local available=$(stat -f -c '%a' ${SCRATCH_MNT})
+ local total=$(stat -f -c '%b' ${SCRATCH_MNT})
+ echo $((100 - (100 * ${available}) / ${total} ))
+}
+
+
+_delete_random_file() {
+ local to_delete=$(find ${SCRATCH_MNT} -type f | shuf | head -1)
+ rm $to_delete
+ sync ${SCRATCH_MNT}
+}
+
+_get_random_fsz() {
+ local r=$RANDOM
+ echo $(( ${min_fsz} + (${max_fsz} - ${min_fsz}) * (${r} % 100) / 100 ))
+}
+
+_direct_fillup () {
+ while [ $(_used_percent) -lt $fill_percent ]; do
+ local fsz=$(_get_random_fsz)
+
+ _create_file $seq $fsz "oflag=direct conv=fsync"
+ seq=$((${seq} + 1))
+ done
+}
+
+_mixed_write_delete() {
+ local dd_extra=$1
+ local total_M=$(_total_M)
+ local to_write_M=$(( ${overwrite_percentage} * ${total_M} / 100 ))
+ local written_M=0
+
+ while [ $written_M -lt $to_write_M ]; do
+ if [ $(_used_percent) -lt $fill_percent ]; then
+ local fsz=$(_get_random_fsz)
+
+ _create_file $seq $fsz "$dd_extra"
+ written_M=$((${written_M} + ${fsz}/${M}))
+ seq=$((${seq} + 1))
+ else
+ _delete_random_file
+ fi
+ done
+}
+
+seed=$RANDOM
+RANDOM=$seed
+echo "Running test with seed=$seed" >>$seqres.full
+
+_scratch_mkfs_sized $((8 * 1024 * 1024 * 1024)) >>$seqres.full
+_scratch_mount
+
+echo "Starting fillup using direct IO"
+_direct_fillup
+
+echo "Starting mixed write/delete test using direct IO"
+_mixed_write_delete "oflag=direct"
+
+echo "Starting mixed write/delete test using buffered IO"
+_mixed_write_delete ""
+
+echo "Syncing"
+sync ${SCRATCH_MNT}/*
+
+echo "Done, all good"
+
+# success, all done
+status=0
+exit
diff --git a/tests/generic/744.out b/tests/generic/744.out
new file mode 100644
index 000000000000..b40c2f43108e
--- /dev/null
+++ b/tests/generic/744.out
@@ -0,0 +1,6 @@
+QA output created by 744
+Starting fillup using direct IO
+Starting mixed write/delete test using direct IO
+Starting mixed write/delete test using buffered IO
+Syncing
+Done, all good
--
2.34.1
next reply other threads:[~2024-04-15 11:23 UTC|newest]
Thread overview: 20+ messages / expand[flat|nested] mbox.gz Atom feed top
2024-04-15 11:23 Hans Holmberg [this message]
2024-04-16 9:07 ` [PATCH] generic: add gc stress test Hans Holmberg
2024-04-16 18:54 ` Darrick J. Wong
2024-04-17 12:43 ` Zorro Lang
2024-04-17 13:21 ` Hans Holmberg
2024-04-17 14:06 ` Zorro Lang
2024-04-17 14:45 ` Hans Holmberg
2024-05-08 7:08 ` Hans Holmberg
2024-05-08 8:51 ` Zorro Lang
2024-05-08 9:28 ` Qu Wenruo
2024-05-08 11:02 ` Johannes Thumshirn
2024-05-09 5:43 ` hch
2024-05-09 9:42 ` Zorro Lang
2024-05-09 12:54 ` hch
2024-05-10 3:21 ` Zorro Lang
2024-05-11 13:08 ` Hans Holmberg
2024-05-12 16:54 ` Johannes Thumshirn
2024-05-12 16:56 ` Johannes Thumshirn
2024-05-13 7:33 ` Qu Wenruo
2024-05-14 8:02 ` Hans Holmberg
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20240415112259.21760-1-hans.holmberg@wdc.com \
--to=hans.holmberg@wdc.com \
--cc=Damien.LeMoal@wdc.com \
--cc=Johannes.Thumshirn@wdc.com \
--cc=Matias.Bjorling@wdc.com \
--cc=Naohiro.Aota@wdc.com \
--cc=fstests@vger.kernel.org \
--cc=hch@lst.de \
--cc=linux-btrfs@vger.kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).