From: Hannes Reinecke <hare@suse.de>
To: Sagi Grimberg <sagi@grimberg.me>,
Hannes Reinecke <hare@kernel.org>, Jens Axboe <axboe@kernel.dk>
Cc: Keith Busch <kbusch@kernel.org>, Christoph Hellwig <hch@lst.de>,
linux-nvme@lists.infradead.org, linux-block@vger.kernel.org
Subject: Re: [PATCH RFC 0/2] block,nvme: latency-based I/O scheduler
Date: Thu, 28 Mar 2024 12:32:05 +0100 [thread overview]
Message-ID: <ef07c956-23cd-4de4-8a4c-c796c743d8f1@suse.de> (raw)
In-Reply-To: <5cade4b4-f19f-422d-ab93-bc853b1563d1@grimberg.me>
On 3/28/24 11:38, Sagi Grimberg wrote:
>
>
> On 26/03/2024 17:35, Hannes Reinecke wrote:
>> Hi all,
>>
>> there had been several attempts to implement a latency-based I/O
>> scheduler for native nvme multipath, all of which had its issues.
>>
>> So time to start afresh, this time using the QoS framework
>> already present in the block layer.
>> It consists of two parts:
>> - a new 'blk-nodelat' QoS module, which is just a simple per-node
>> latency tracker
>> - a 'latency' nvme I/O policy
>>
>> Using the 'tiobench' fio script I'm getting:
>> WRITE: bw=531MiB/s (556MB/s), 33.2MiB/s-52.4MiB/s
>> (34.8MB/s-54.9MB/s), io=4096MiB (4295MB), run=4888-7718msec
>> WRITE: bw=539MiB/s (566MB/s), 33.7MiB/s-50.9MiB/s
>> (35.3MB/s-53.3MB/s), io=4096MiB (4295MB), run=5033-7594msec
>> READ: bw=898MiB/s (942MB/s), 56.1MiB/s-75.4MiB/s
>> (58.9MB/s-79.0MB/s), io=4096MiB (4295MB), run=3397-4560msec
>> READ: bw=1023MiB/s (1072MB/s), 63.9MiB/s-75.1MiB/s
>> (67.0MB/s-78.8MB/s), io=4096MiB (4295MB), run=3408-4005msec
>>
>> for 'round-robin' and
>>
>> WRITE: bw=574MiB/s (601MB/s), 35.8MiB/s-45.5MiB/s
>> (37.6MB/s-47.7MB/s), io=4096MiB (4295MB), run=5629-7142msec
>> WRITE: bw=639MiB/s (670MB/s), 39.9MiB/s-47.5MiB/s
>> (41.9MB/s-49.8MB/s), io=4096MiB (4295MB), run=5388-6408msec
>> READ: bw=1024MiB/s (1074MB/s), 64.0MiB/s-73.7MiB/s
>> (67.1MB/s-77.2MB/s), io=4096MiB (4295MB), run=3475-4000msec
>> READ: bw=1013MiB/s (1063MB/s), 63.3MiB/s-72.6MiB/s
>> (66.4MB/s-76.2MB/s), io=4096MiB (4295MB), run=3524-4042msec
>> for 'latency' with 'decay' set to 10.
>> That's on a 32G FC testbed running against a brd target,
>> fio running with 16 thread.
>
> Can you quantify the improvement? Also, the name latency suggest
> that latency should be improved no?
>
'latency' refers to 'latency-based' I/O scheduler, ie it selects
the path with the least latency. It does not necessarily _improve_
the latency. Eg for truly symmetric fabrics it doesn't.
It _does_ improve matters when running on asymmetric fabrics
(eg on a two socket system with two PCI HBAs, each connected to one
socket, or like the example above with one path via 'loop', and the
other via 'tcp' and address '127.0.0.1').
And, of course, if you have congested fabrics, where it should be
able to direct I/O to the least congested path.
But I'll see to extract the latency numbers, too.
What I really wanted to show is that we _can_ track latency without
harming performance.
Cheers,
Hannes
prev parent reply other threads:[~2024-03-28 11:32 UTC|newest]
Thread overview: 7+ messages / expand[flat|nested] mbox.gz Atom feed top
2024-03-26 15:35 [PATCH RFC 0/2] block,nvme: latency-based I/O scheduler Hannes Reinecke
2024-03-26 15:35 ` [PATCH 1/2] block: track per-node I/O latency Hannes Reinecke
2024-03-27 18:03 ` kernel test robot
2024-03-27 20:59 ` kernel test robot
2024-03-26 15:35 ` [PATCH 2/2] nvme: add 'latency' iopolicy Hannes Reinecke
2024-03-28 10:38 ` [PATCH RFC 0/2] block,nvme: latency-based I/O scheduler Sagi Grimberg
2024-03-28 11:32 ` Hannes Reinecke [this message]
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=ef07c956-23cd-4de4-8a4c-c796c743d8f1@suse.de \
--to=hare@suse.de \
--cc=axboe@kernel.dk \
--cc=hare@kernel.org \
--cc=hch@lst.de \
--cc=kbusch@kernel.org \
--cc=linux-block@vger.kernel.org \
--cc=linux-nvme@lists.infradead.org \
--cc=sagi@grimberg.me \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).