From: Jens Axboe <axboe@kernel.dk>
To: Pavel Begunkov <asml.silence@gmail.com>,
Dylan Yudaken <dyudaken@gmail.com>
Cc: io-uring@vger.kernel.org
Subject: Re: [PATCH 6/8] io_uring/net: support multishot for send
Date: Thu, 29 Feb 2024 08:42:18 -0700 [thread overview]
Message-ID: <d980459c-6070-4b9a-9808-a8497df44ef9@kernel.dk> (raw)
In-Reply-To: <a61390aa-01a4-4511-a24a-c3373110a28a@kernel.dk>
On 2/28/24 6:46 PM, Jens Axboe wrote:
> On 2/28/24 4:49 PM, Jens Axboe wrote:
>> On 2/28/24 10:28 AM, Jens Axboe wrote:
>>> When I have some time I can do add the append case, or feel free to do
>>> that yourself, and I can run some testing with that too.
>>
>> I did a quick hack to add the append mode, and by default we get roughly
>> ring_size / 2 number of appended vecs, which I guess is expected.
>> There's a few complications there:
>>
>> 1) We basically need a per-send data structure at this point. While
>> that's not hard to do, at least for my case I'd need to add that just
>> for this case and everything would now need to do it. Perhaps. You
>> can perhaps steal a few low bits and only do it for sendmsg. But why?
>> Because now you have multiple buffer IDs in a send completion, and
>> you need to be able to unravel it. If we always receive and send in
>> order, then it'd always been contiguous, which I took advantage of.
>> Not a huge deal, just mentioning some of the steps I had to go
>> through.
>>
>> 2) The iovec. Fine if you have the above data structure, as you can just
>> alloc/realloc -> free when done. I just took the easy path and made
>> the iovec big enough (eg ring size).
>>
>> Outside of that, didn't need to make a lot of changes:
>>
>> 1 file changed, 39 insertions(+), 17 deletions(-)
>>
>> Performance is great, because we get to pass in N (in my case ~65)
>> packets per send. No more per packet locking. Which I do think
>> highlights that we can do better on the multishot send/sendmsg by
>> grabbing buffers upfront and passing them in in one go rather than
>> simply loop around calling tcp_sendmsg_locked() for each one.
>
> In terms of absolute numbers, previous best times were multishot send,
> with ran in 3125 usec. Using either the above approach, or a hacked up
> version of multishot send that uses provided buffers and bundles them
> into one send (ala sendmsg), the runtimes are within 1% of each other
> and too close to call. But the runtime is around 2320, or aroudn 25%
> faster than doing one issue at the time.
>
> This is using the same small packet size of 32 bytes. Just did the
> bundled send multishot thing to test, haven't tested more than that so
> far.
Update: I had a bug in the sendmsg with multiple vecs, it did not have a
single msg inflight at the same time, it could be multiple. Which
obviously can't work. That voids the sendmsg append results for now.
Will fiddle with it a bit and get it working, and post the full results
when I have them.
--
Jens Axboe
next prev parent reply other threads:[~2024-02-29 15:42 UTC|newest]
Thread overview: 34+ messages / expand[flat|nested] mbox.gz Atom feed top
2024-02-25 0:35 [PATCHSET v3 0/8] Support for provided buffers for send Jens Axboe
2024-02-25 0:35 ` [PATCH 1/8] io_uring/net: unify how recvmsg and sendmsg copy in the msghdr Jens Axboe
2024-02-26 14:41 ` Pavel Begunkov
2024-02-26 15:03 ` Jens Axboe
2024-02-25 0:35 ` [PATCH 2/8] net: remove {revc,send}msg_copy_msghdr() from exports Jens Axboe
2024-02-25 0:35 ` [PATCH 3/8] io_uring/net: add provided buffer support for IORING_OP_SEND Jens Axboe
2024-02-25 0:35 ` [PATCH 4/8] io_uring/net: add provided buffer support for IORING_OP_SENDMSG Jens Axboe
2024-02-25 0:35 ` [PATCH 5/8] io_uring/kbuf: flag request if buffer pool is empty after buffer pick Jens Axboe
2024-02-25 0:35 ` [PATCH 6/8] io_uring/net: support multishot for send Jens Axboe
2024-02-26 10:47 ` Dylan Yudaken
2024-02-26 13:38 ` Jens Axboe
2024-02-26 14:02 ` Dylan Yudaken
2024-02-26 14:27 ` Jens Axboe
2024-02-26 14:36 ` Pavel Begunkov
2024-02-26 15:16 ` Jens Axboe
2024-02-26 15:41 ` Pavel Begunkov
2024-02-26 19:11 ` Jens Axboe
2024-02-26 19:21 ` Pavel Begunkov
2024-02-26 20:12 ` Jens Axboe
2024-02-26 20:51 ` Pavel Begunkov
2024-02-26 21:27 ` Jens Axboe
2024-02-28 12:39 ` Pavel Begunkov
2024-02-28 17:28 ` Jens Axboe
2024-02-28 23:49 ` Jens Axboe
2024-02-29 1:46 ` Jens Axboe
2024-02-29 15:42 ` Jens Axboe [this message]
2024-02-26 19:31 ` Dylan Yudaken
2024-02-26 19:49 ` Jens Axboe
2024-02-25 0:35 ` [PATCH 7/8] io_uring/net: support multishot for sendmsg Jens Axboe
2024-02-25 0:35 ` [PATCH 8/8] io_uring/net: set MSG_MORE if we're doing multishot send and have more Jens Axboe
2024-02-26 10:59 ` Dylan Yudaken
2024-02-26 13:42 ` Jens Axboe
2024-02-26 14:24 ` Pavel Begunkov
2024-02-26 14:52 ` Jens Axboe
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=d980459c-6070-4b9a-9808-a8497df44ef9@kernel.dk \
--to=axboe@kernel.dk \
--cc=asml.silence@gmail.com \
--cc=dyudaken@gmail.com \
--cc=io-uring@vger.kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).