From: zhenwei pi <pizhenwei@bytedance.com>
To: Raphael Norwitz <raphael.norwitz@nutanix.com>
Cc: Stefan Hajnoczi <stefanha@redhat.com>,
"parav@nvidia.com" <parav@nvidia.com>,
"Michael S. Tsirkin" <mst@redhat.com>,
Jason Wang <jasowang@redhat.com>,
"virtio-comment@lists.oasis-open.org"
<virtio-comment@lists.oasis-open.org>,
"houp@yusur.tech" <houp@yusur.tech>,
"zhouhuaping.san@bytedance.com" <zhouhuaping.san@bytedance.com>,
"xinhao.kong@duke.edu" <xinhao.kong@duke.edu>
Subject: [virtio-comment] Re: Re: [PATCH v5 3/7] transport-fabrics: introduce Virtio-oF Protocol Data Unit
Date: Tue, 12 Dec 2023 17:19:15 +0800 [thread overview]
Message-ID: <61d180ed-d4c2-4764-aea4-f8ac5fa0c007@bytedance.com> (raw)
In-Reply-To: <EA77A282-0A38-4B97-B6BC-34449CDEAC9D@nutanix.com>
On 12/12/23 14:22, Raphael Norwitz wrote:
> One suggestion I didn’t think of in my prior review.
>
>> On Oct 23, 2023, at 6:46 AM, zhenwei pi <pizhenwei@bytedance.com> wrote:
>>
>> Introduce Virtio-oF PDU for Virtio-oF queue, add Stream Data Transfers
>> and Keyed Data Transfers mechanism.
>>
>> Signed-off-by: zhenwei pi <pizhenwei@bytedance.com>
>> —
> <snip>
>> +\begin{lstlisting}
>> +struct virtio_of_keyed_desc {
>> + /* the remote address of data */
>> + le64 addr;
>> + /* the length of data */
>> + le32 length;
>> + /* the key to access the remote data */
>> + le32 key;
>> +};
>> +\end{lstlisting}
>
> Going off my comment here: https://lists.oasis-open.org/archives/virtio-comment/202307/msg00253.html
>
> You could probably save some space space in virtio_of_keyed_desc with a protocol extension where you can pre-register some number of MRs/keys with the target and reference them by index. For example, you could save 16 bits per descriptor, lowering the total memory footprint by 1/8 by doing something like:
>
> struct virtio_of_small_keyed_desc {
> /* the remote address of data */
> le64 addr;
> /* the length of data */
> le32 length;
> /* the mr_id which maps to a key (and possibly a base address) on the target size */
> le16 mr_hint;
> };
>
> If you could bound the size of each MR (to, say 4GB) it could make sense to send a 32 bit offset into the MR and then re-compute the address on the target side.
>
In my imagination, for example, the target reports the max Keyed desc as 4:
1, the initiator side has less segments(Ex, 3), the initiator just sends
3 Keyed desc.
2, the initiator side has more segments(Ex, 6), the initiator side has
to allocate a large memory, copy the suitable segments to the allocated
memory(or copy from).
Ex, OUT[0] 16B, OUT[1] 4K, OUT[2] 4K, OUT[3] 16K, OUT[4] 8K, IN[5] 1B
then the initiator side needs allocate 8193 memory, copies OUT[0],
OUT[1] and OUT[2] into new memory, finally sends 4 Keyed desc.
From my knowledge, most virtio devices usually use fixed segments(not a
hard limitation), virtio-blk also reports max_seg in config. So I
suppose the target side could report reasonable value to avoid memory copy.
--
zhenwei pi
This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.
Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
next prev parent reply other threads:[~2023-12-12 9:24 UTC|newest]
Thread overview: 13+ messages / expand[flat|nested] mbox.gz Atom feed top
2023-10-23 10:46 [virtio-comment] [PATCH v5 0/7] Introduce Virtio Over Fabrics zhenwei pi
2023-10-23 10:46 ` [virtio-comment] [PATCH v5 1/7] transport-fabrics: introduce Virtio Over Fabrics overview zhenwei pi
2023-10-23 10:46 ` [virtio-comment] [PATCH v5 2/7] transport-fabrics: introduce Virtio-oF Qualified Name zhenwei pi
2023-10-23 10:46 ` [virtio-comment] [PATCH v5 3/7] transport-fabrics: introduce Virtio-oF Protocol Data Unit zhenwei pi
2023-12-12 6:22 ` [virtio-comment] " Raphael Norwitz
2023-12-12 9:19 ` zhenwei pi [this message]
2023-10-23 10:46 ` [virtio-comment] [PATCH v5 4/7] transport-fabrics: introduce command set zhenwei pi
2023-10-23 10:46 ` [virtio-comment] [PATCH v5 5/7] transport-fabrics: introduce transport binding zhenwei pi
2023-10-23 10:46 ` [virtio-comment] [PATCH v5 6/7] transport-fabrics: add device initialization zhenwei pi
2023-10-23 10:46 ` [virtio-comment] [PATCH v5 7/7] transport-fabrics: introduce keyed eager buffers zhenwei pi
2023-11-17 3:43 ` [virtio-comment] Re: [PATCH v5 0/7] Introduce Virtio Over Fabrics zhenwei pi
2023-12-12 6:23 ` Raphael Norwitz
2023-12-12 8:36 ` [virtio-comment] " zhenwei pi
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=61d180ed-d4c2-4764-aea4-f8ac5fa0c007@bytedance.com \
--to=pizhenwei@bytedance.com \
--cc=houp@yusur.tech \
--cc=jasowang@redhat.com \
--cc=mst@redhat.com \
--cc=parav@nvidia.com \
--cc=raphael.norwitz@nutanix.com \
--cc=stefanha@redhat.com \
--cc=virtio-comment@lists.oasis-open.org \
--cc=xinhao.kong@duke.edu \
--cc=zhouhuaping.san@bytedance.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).