From: Jens Axboe <axboe@kernel.dk>
To: io-uring@vger.kernel.org
Cc: asml.silence@gmail.com, dyudaken@gmail.com, dw@davidwei.uk
Subject: Re: [PATCHSET RFC 0/7] Send and receive bundles
Date: Sun, 10 Mar 2024 12:15:33 -0600 [thread overview]
Message-ID: <e557e37c-98f4-42a4-9a89-a0dd46b123de@kernel.dk> (raw)
In-Reply-To: <20240308235045.1014125-1-axboe@kernel.dk>
[-- Attachment #1: Type: text/plain, Size: 3274 bytes --]
On 3/8/24 4:34 PM, Jens Axboe wrote:
> Hi,
>
> I went back to the drawing board a bit on the send multishot, and this
> is what came out.
>
> First support was added for provided buffers for send. This works like
> provided buffers for recv/recvmsg, and the intent here to use the buffer
> ring queue as an outgoing sequence for sending.
>
> But the real meat is adding support for picking multiple buffers at the
> time, what I dubbed "bundles" here. Rather than just pick a single buffer
> for send, it can pick a bunch of them and send them in one go. The idea
> here is that the expensive part of a request is not the sqe issue, it's
> the fact that we have to do each buffer separately. That entails calling
> all the way down into the networking stack, locking the socket, checking
> what needs doing afterwards (like flushing the backlog), unlocking the
> socket, etc. If we have an outgoing send queue, then pick what buffers
> we have (up to a certain cap), and pass them to the networking stack in
> one go.
>
> Bundles must be used with provided buffers, obviously. At completion
> time, they pass the starting buffer ID in cqe->flags, like any other
> provided buffer completion. cqe->res is the TOTAL number of bytes sent,
> so it's up to the application to iterate buffers to figure out how many
> completed. This part is trivial. I'll push the proxy changes out soon,
> just need to cleanup them up as I did the sendmsg bundling too and would
> love to compare.
>
> With that in place, I added support for recv for bundles as well. Exactly
> the same as the send side - if we have a known amount of data pending,
> pick enough buffers to satisfy the receive and post a single completion
> for that round. Buffer ID in cqe->flags, cqe->res is the total number of
> buffers sent. Receive can be used with multishot as well - fire off one
> multishot recv, and keep getting big completions. Unfortunately, recvmsg
> multishot is just not as efficient as recv, as it carries additional
> data that needs copying. recv multishot with bundles provide a good
> alternative to recvmsg, if all you need is more than one range of data.
> I'll compare these too soon as well.
>
> This is obviously a bigger win for smaller packets than for large ones,
> as the overall cost of entering sys_sendmsg/sys_recvmsg() in terms of
> throughput decreases as the packet size increases. For the extreme end,
> using 32b packets, performance increases substantially. Runtime for
> proxying 32b packets between three machines on a 10G link for the test:
>
> Send ring: 3462 msec 1183Mbit
> Send ring + bundles 844 msec 4853Mbit
>
> and bundles reach 100% bandwidth at 80b of packet size, compared to send
> ring alone needing 320b to reach 95% of bandwidth (I didn't redo that
> test so don't have the 100% number).
Re-did all the numbers, see attached graph. tldr is that send bundles OR
sendmsg are by far the fastest, they hit line rate very quickly. This is
expected as both of these send methods can pack more than a single
packet into a send operation, reducing the cost of the smaller payloads.
Looking at profiles, sendmsg does use ~3.5% more CPU for the same work.
Which is also expected, it needs to do a bit more work to accomplish the
same.
--
Jens Axboe
[-- Attachment #2: io_uring-10G-test.png --]
[-- Type: image/png, Size: 64083 bytes --]
prev parent reply other threads:[~2024-03-10 18:15 UTC|newest]
Thread overview: 9+ messages / expand[flat|nested] mbox.gz Atom feed top
2024-03-08 23:34 [PATCHSET RFC 0/7] Send and receive bundles Jens Axboe
2024-03-08 23:34 ` [PATCH 1/7] io_uring/net: add generic multishot retry helper Jens Axboe
2024-03-08 23:34 ` [PATCH 2/7] io_uring/net: add provided buffer support for IORING_OP_SEND Jens Axboe
2024-03-08 23:34 ` [PATCH 3/7] io_uring/kbuf: add helpers for getting/peeking multiple buffers Jens Axboe
2024-03-08 23:34 ` [PATCH 4/7] io_uring/net: switch io_send() and io_send_zc() to using io_async_msghdr Jens Axboe
2024-03-08 23:34 ` [PATCH 5/7] io_uring/net: support bundles for send Jens Axboe
2024-03-08 23:34 ` [PATCH 6/7] io_uring/net: switch io_recv() to using io_async_msghdr Jens Axboe
2024-03-08 23:34 ` [PATCH 7/7] io_uring/net: support bundles for recv Jens Axboe
2024-03-10 18:15 ` Jens Axboe [this message]
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=e557e37c-98f4-42a4-9a89-a0dd46b123de@kernel.dk \
--to=axboe@kernel.dk \
--cc=asml.silence@gmail.com \
--cc=dw@davidwei.uk \
--cc=dyudaken@gmail.com \
--cc=io-uring@vger.kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).