Re: [PATCH v8 00/10] Introduce VDUSE - vDPA Device in Userspace

From: Jason Wang <jasowang@redhat.com>
To: "Liu, Xiaodong" <xiaodong.liu@intel.com>,
	Xie Yongji <xieyongji@bytedance.com>,
	"mst@redhat.com" <mst@redhat.com>,
	"stefanha@redhat.com" <stefanha@redhat.com>,
	"sgarzare@redhat.com" <sgarzare@redhat.com>,
	"parav@nvidia.com" <parav@nvidia.com>,
	"hch@infradead.org" <hch@infradead.org>,
	"christian.brauner@canonical.com"
	<christian.brauner@canonical.com>,
	"rdunlap@infradead.org" <rdunlap@infradead.org>,
	"willy@infradead.org" <willy@infradead.org>,
	"viro@zeniv.linux.org.uk" <viro@zeniv.linux.org.uk>,
	"axboe@kernel.dk" <axboe@kernel.dk>,
	"bcrl@kvack.org" <bcrl@kvack.org>,
	"corbet@lwn.net" <corbet@lwn.net>,
	"mika.penttila@nextfour.com" <mika.penttila@nextfour.com>,
	"dan.carpenter@oracle.com" <dan.carpenter@oracle.com>,
	"joro@8bytes.org" <joro@8bytes.org>,
	"gregkh@linuxfoundation.org" <gregkh@linuxfoundation.org>
Cc: "kvm@vger.kernel.org" <kvm@vger.kernel.org>,
	"netdev@vger.kernel.org" <netdev@vger.kernel.org>,
	"linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
	"virtualization@lists.linux-foundation.org"
	<virtualization@lists.linux-foundation.org>,
	"iommu@lists.linux-foundation.org"
	<iommu@lists.linux-foundation.org>,
	"songmuchun@bytedance.com" <songmuchun@bytedance.com>,
	"linux-fsdevel@vger.kernel.org" <linux-fsdevel@vger.kernel.org>
Subject: Re: [PATCH v8 00/10] Introduce VDUSE - vDPA Device in Userspace
Date: Tue, 29 Jun 2021 12:10:51 +0800	[thread overview]
Message-ID: <41cc419e-48b5-6755-0cb0-9033bd1310e4@redhat.com> (raw)
In-Reply-To: <BYAPR11MB2662FFF6140A4C634648BB2E8C039@BYAPR11MB2662.namprd11.prod.outlook.com>

在 2021/6/28 下午1:54, Liu, Xiaodong 写道:
>> Several issues:
>>
>> - VDUSE needs to limit the total size of the bounce buffers (64M if I was not
>> wrong). Does it work for SPDK?
> Yes, Jason. It is enough and works for SPDK.
> Since it's a kind of bounce buffer mainly for in-flight IO, so limited size like
> 64MB is enough.

Ok.

>
>> - VDUSE can use hugepages but I'm not sure we can mandate hugepages (or we
>> need introduce new flags for supporting this)
> Same with your worry, I'm afraid too that it is a hard for a kernel module
> to directly preallocate hugepage internal.
> What I tried is that:
> 1. A simple agent daemon (represents for one device)  `preallocates` and maps
>      dozens of 2MB hugepages (like 64MB) for one device.
> 2. The daemon passes its mapping addr&len and hugepage fd to kernel
>      module through created IOCTL.
> 3. Kernel module remaps the hugepages inside kernel.

Such model should work, but the main "issue" is that it introduce  
overheads in the case of vhost-vDPA.

Note that in the case of vhost-vDPA, we don't use bounce buffer, the  
userspace pages were shared directly.

And since DMA is not done per page, it prevents us from using tricks  
like vm_insert_page() in those cases.

> 4. Vhost user target gets and maps hugepage fd from kernel module
>      in vhost-user msg through Unix Domain Socket cmsg.
> Then kernel module and target map on the same hugepage based
> bounce buffer for in-flight IO.
>
> If there is one option in VDUSE to map userspace preallocated memory, then
> VDUSE should be able to mandate it even it is hugepage based.
>

As above, this requires some kind of re-design since VDUSE depends on  
the model of mmap(MAP_SHARED) instead of umem registering.

Thanks

_______________________________________________
iommu mailing list
iommu@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/iommu