From: Coly Li <colyli@suse.de>
To: Eric Wheeler <bcache@lists.ewheeler.net>
Cc: Nikhil Kshirsagar <nkshirsagar@gmail.com>, linux-bcache@vger.kernel.org
Subject: Re: bcache I/O performance tests on 5.15.0-40-generic
Date: Thu, 28 Jul 2022 23:51:04 +0800 [thread overview]
Message-ID: <1E966D2E-1B9E-46DD-BF79-A3FEDAF227ED@suse.de> (raw)
In-Reply-To: <37577c49-6d0-e5f4-2ea3-51128526526e@ewheeler.net>
> 2022年7月6日 04:49，Eric Wheeler <bcache@lists.ewheeler.net> 写道：
>
> On Sat, 25 Jun 2022, Coly Li wrote:
>>> 2022年6月25日 14:29，Nikhil Kshirsagar <nkshirsagar@gmail.com> 写道：
>>>
>>> Hello,
>>>
>>> I've been doing some performance tests of bcache on 5.15.0-40-generic.
>>>
>>> The baseline figures for the fast and slow disk for random writes are
>>> consistent at around 225MiB/s and 3046KiB/s.
>>>
>>> But the bcache results inexplicably drop sometimes to 10Mib/s, for
>>> random write test using fio like this -
>>>
>>> fio --rw=randwrite --size=1G --ioengine=libaio --direct=1
>>> --gtod_reduce=1 --iodepth=128 --bs=4k --name=MY_TEST1
>>>
>>> WRITE: bw=168MiB/s (176MB/s), 168MiB/s-168MiB/s (176MB/s-176MB/s),
>>> io=1024MiB (1074MB), run=6104-6104msec
>>> WRITE: bw=283MiB/s (297MB/s), 283MiB/s-283MiB/s (297MB/s-297MB/s),
>>> io=1024MiB (1074MB), run=3621-3621msec
>>> WRITE: bw=10.3MiB/s (10.9MB/s), 10.3MiB/s-10.3MiB/s
>>> (10.9MB/s-10.9MB/s), io=1024MiB (1074MB), run=98945-98945msec
>>> WRITE: bw=8236KiB/s (8434kB/s), 8236KiB/s-8236KiB/s
>>> (8434kB/s-8434kB/s), io=1024MiB (1074MB), run=127317-127317msec
>>> WRITE: bw=9657KiB/s (9888kB/s), 9657KiB/s-9657KiB/s
>>> (9888kB/s-9888kB/s), io=1024MiB (1074MB), run=108587-108587msec
>>> WRITE: bw=4543KiB/s (4652kB/s), 4543KiB/s-4543KiB/s
>>> (4652kB/s-4652kB/s), io=1024MiB (1074MB), run=230819-230819msec
>>>
>>> This seems to happen after 2 runs of 1gb writes (cache disk is 4gb size)
>>>
>>> Some details are here - https://pastebin.com/V9mpLCbY , I will share
>>> the full testing results soon, but just was wondering about this
>>> performance drop for no apparent reason once the cache gets about 50%
>>> full.
>>
>>
>> It seems you are stuck by garbage collection. 4GB cache is small, the
>> garbage collection might be invoked quite frequently. Maybe you can see
>> the output of ’top -H’ to check whether there is kernel thread named
>> bache_gc.
>
> Hi Nikhil,
>
> Do you have Mingzhe's GC patch? It might help:
> https://www.spinics.net/lists/linux-bcache/msg11185.html
>
> Coli, did Mingzhe's patch get into your testing tree? It looks like it
> could be a good addition to bcache.
No. This patch just reduce the early gc action to make IO faster, and accumulates a large gc action finally to cause more lower I/O period.
The later larger gc may cause longer unpredictable time I/O stuck, which doesn’t follow current bcache behavior.
It may be helpful for some I/O workloads, but for continuous heavy I/O loads, it won’t help too much.
Coly Li
prev parent reply other threads:[~2022-07-28 15:51 UTC|newest]
Thread overview: 9+ messages / expand[flat|nested] mbox.gz Atom feed top
2022-06-25 6:29 bcache I/O performance tests on 5.15.0-40-generic Nikhil Kshirsagar
2022-06-25 12:08 ` Coly Li
2022-06-30 4:24 ` Nikhil Kshirsagar
[not found] ` <CAC6jXv0yOZ98XqG=quDcONuZ9ggqK4doM8EzVTc=Sk1m-H=_Xw@mail.gmail.com>
2022-06-30 6:49 ` Coly Li
[not found] ` <CAC6jXv2u_0s-hvj4J4gurfoxgKNHFcHYq9F8cfcf-s6oG+pU+Q@mail.gmail.com>
2022-06-30 7:36 ` Coly Li
2022-06-30 7:39 ` Nikhil Kshirsagar
2022-06-30 7:47 ` Coly Li
2022-07-05 20:49 ` Eric Wheeler
2022-07-28 15:51 ` Coly Li [this message]
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=1E966D2E-1B9E-46DD-BF79-A3FEDAF227ED@suse.de \
--to=colyli@suse.de \
--cc=bcache@lists.ewheeler.net \
--cc=linux-bcache@vger.kernel.org \
--cc=nkshirsagar@gmail.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).