From: "Zeng, Oak" <oak.zeng@intel.com>
To: "Christian König" <christian.koenig@amd.com>,
"Thomas Hellström" <thomas.hellstrom@linux.intel.com>,
"Daniel Vetter" <daniel@ffwll.ch>,
"Dave Airlie" <airlied@redhat.com>
Cc: "Brost, Matthew" <matthew.brost@intel.com>,
"rcampbell@nvidia.com" <rcampbell@nvidia.com>,
"apopple@nvidia.com" <apopple@nvidia.com>,
Felix Kuehling <felix.kuehling@amd.com>,
"Welty, Brian" <brian.welty@intel.com>,
"Shah, Ankur N" <ankur.n.shah@intel.com>,
"dri-devel@lists.freedesktop.org"
<dri-devel@lists.freedesktop.org>,
"jglisse@redhat.com" <jglisse@redhat.com>,
"Ghimiray, Himal Prasad" <himal.prasad.ghimiray@intel.com>,
"Gupta, saurabhg" <saurabhg.gupta@intel.com>,
"Bommu, Krishnaiah" <krishnaiah.bommu@intel.com>,
"Vishwanathapura,
Niranjana" <niranjana.vishwanathapura@intel.com>,
"intel-xe@lists.freedesktop.org" <intel-xe@lists.freedesktop.org>,
Danilo Krummrich <dakr@redhat.com>
Subject: RE: Making drm_gpuvm work across gpu devices
Date: Tue, 30 Jan 2024 22:29:08 +0000 [thread overview]
Message-ID: <SA1PR11MB6991E4CF9949916C6290C232927D2@SA1PR11MB6991.namprd11.prod.outlook.com> (raw)
In-Reply-To: <801c3357-d49b-4505-a1a4-ef2f36ba63e0@amd.com>
[-- Attachment #1: Type: text/plain, Size: 17584 bytes --]
Hi Christian,
Nvidia Nouveau driver uses exactly the same concept of SVM with HMM, GPU address in the same process is exactly the same with CPU virtual address. It is already in upstream Linux kernel. We Intel just follow the same direction for our customers. Why we are not allowed?
From: Christian König <christian.koenig@amd.com>
Sent: Tuesday, January 30, 2024 3:40 AM
To: Zeng, Oak <oak.zeng@intel.com>; Thomas Hellström <thomas.hellstrom@linux.intel.com>; Daniel Vetter <daniel@ffwll.ch>; Dave Airlie <airlied@redhat.com>
Cc: Brost, Matthew <matthew.brost@intel.com>; Felix Kuehling <felix.kuehling@amd.com>; Welty, Brian <brian.welty@intel.com>; dri-devel@lists.freedesktop.org; Ghimiray, Himal Prasad <himal.prasad.ghimiray@intel.com>; Bommu, Krishnaiah <krishnaiah.bommu@intel.com>; Gupta, saurabhg <saurabhg.gupta@intel.com>; Vishwanathapura, Niranjana <niranjana.vishwanathapura@intel.com>; intel-xe@lists.freedesktop.org; Danilo Krummrich <dakr@redhat.com>
Subject: Re: Making drm_gpuvm work across gpu devices
Am 30.01.24 um 01:21 schrieb Zeng, Oak:
The example you used to prove that KFD is a design failure, is against *any* design which utilize system allocator and hmm. The way that one proxy process running on host to handle many guest processes, doesn’t fit into the concept of “share address space b/t cpu and gpu”. The shared address space has to be within one process. Your proxy process represent many guest processes. It is a fundamental conflict.
Also your userptr proposal does’t solve this problem either:
Imagine you have guest process1 mapping CPU address range A…B to GPU address range C…D
And you have guest process 2 mapping CPU address range A…B to GPU address range C…D, since process 1 and 2 are two different process, it is legal for process 2 to do the exact same mapping.
Now when gpu shader access address C…D, a gpu page fault happens, what does your proxy process do? Which guest process will this fault be directed to and handled? Except you have extra information/API to tell proxy process and GPU HW, there is no way to figure out.
Well yes, as far as I can see the fundamental design issue in the KFD is that it ties together CPU and GPU address space. That came from the implementation using the ATS/PRI feature to access the CPU address space from the GPU.
If you don't do ATS/PRI then you don't have that restriction and you can do as many GPU address spaces per CPU process as you want. This is just how the hw works.
So in your example above when you have multiple mappings for the range A..B you also have multiple GPU address spaces and so can distinct where the page fault is coming from just by looking at the source of it. All you then need is userfaultfd() to forward the fault to the client and you are pretty much done.
Compared to the shared virtual address space concept of HMM, the userptr design is nothing new except it allows CPU and GPU to use different address to access the same object. If you replace above C…D with A…B, above description becomes a description of the “problem” of HMM/shared virtual address design.
Both design has the same difficulty with your example of the special virtualization environment setup.
As said, we spent effort scoped the userptr solution some time ago. The problem we found enabling userptr with migration were:
1. The user interface of userptr is not as convenient as system allocator. With the userptr solution, user need to call userptr_ioctl and vm_bind for *every* single cpu pointer that he want to use in a gpu program. While with system allocator, programmer just use any cpu pointer directly in gpu program without any extra driver ioctls.
And I think exactly that is questionable. Why not at least call it for the whole address space once during initialization?
> We don’t see the real benefit of using a different Gpu address C…D than the A..B, except you can prove my above reasoning is wrong. In most use cases, you can make GPU C…D == CPU A…B, why bother then?
Because there are cases where this isn't true. We just recently ran into exactly that use case with a customer. It might be that you will never need this, but again the approach should generally be that the kernel exposes the hardware features and as far as I can see the hardware can do this.
And apart from those use cases there is also another good reason for this: CPU are going towards 5 level of page tables and GPUs are lacking behind. It's not unrealistic to run into cases where you can only mirror parts of the CPU address space into the GPU address space because of hardware restrictions. And in this case you absolutely do want the flexibility to have different address ranges.
> Looked into implementation details, since hmm fundamentally assume a shared virtual address space b/t cpu and device, for the userptr solution to leverage hmm, you need perform address space conversion every time you calls into hmm functions.
Correct, but that is trivial. I mean we do nothing else with VMAs mapping into the address space of files on the CPU either.
Which is by the way a good analogy. The CPU address space consists of anonymous memory and file mappings, where the later covers both real files on a file system as well as devices.
The struct address_space in the Linux kernel for example describes a file address space and not the CPU address space because the later is just a technical tool to form an execution environment which can access the former.
With GPUs it's pretty much the same. You have mappings which can be backed by CPU address space using functionalities like HMM as well as buffer objects created directly through device drivers.
In summary, GPU device is just a piece of HW to accelerate your CPU program.
Well exactly that's not how I see it. CPU accelerators are extensions like SSE, AVX, FPUs etc... GPU are accelerators attached as I/O devices.
And that GPUs are separate to the CPU is a benefit which gives them advantage over CPU based acceleration approaches.
This obviously makes GPUs harder to program and SVM is a method to counter this, but that doesn't make SVM a good design pattern for kernel or device driver interfaces.
If HW allows, it is more convenient to use shared address space b/t cpu and GPU. On old HW (example, no gpu page fault support, or gpu only has a very limited address space), we can disable system allocator/SVM. If you use different address space on modern GPU, why don’t you use different address space on different CPU cores?
Quite simple, modern CPU are homogeneous. From the application point of view they still look more or less the same they did 40 years ago.
GPUs on the other hand look quite a bit different. SVM is now a tool to reduce this difference but it doesn't make the differences in execution environment go away.
And I can only repeat myself that this is actually a good thing, cause otherwise GPUs would loose some of their advantage over CPUs.
Regards,
Christian.
Regards,
Oak
From: dri-devel <dri-devel-bounces@lists.freedesktop.org><mailto:dri-devel-bounces@lists.freedesktop.org> On Behalf Of Christian König
Sent: Monday, January 29, 2024 5:20 AM
To: Zeng, Oak <oak.zeng@intel.com><mailto:oak.zeng@intel.com>; Thomas Hellström <thomas.hellstrom@linux.intel.com><mailto:thomas.hellstrom@linux.intel.com>; Daniel Vetter <daniel@ffwll.ch><mailto:daniel@ffwll.ch>; Dave Airlie <airlied@redhat.com><mailto:airlied@redhat.com>
Cc: Brost, Matthew <matthew.brost@intel.com><mailto:matthew.brost@intel.com>; Felix Kuehling <felix.kuehling@amd.com><mailto:felix.kuehling@amd.com>; Welty, Brian <brian.welty@intel.com><mailto:brian.welty@intel.com>; dri-devel@lists.freedesktop.org<mailto:dri-devel@lists.freedesktop.org>; Ghimiray, Himal Prasad <himal.prasad.ghimiray@intel.com><mailto:himal.prasad.ghimiray@intel.com>; Bommu, Krishnaiah <krishnaiah.bommu@intel.com><mailto:krishnaiah.bommu@intel.com>; Gupta, saurabhg <saurabhg.gupta@intel.com><mailto:saurabhg.gupta@intel.com>; Vishwanathapura, Niranjana <niranjana.vishwanathapura@intel.com><mailto:niranjana.vishwanathapura@intel.com>; intel-xe@lists.freedesktop.org<mailto:intel-xe@lists.freedesktop.org>; Danilo Krummrich <dakr@redhat.com><mailto:dakr@redhat.com>
Subject: Re: Making drm_gpuvm work across gpu devices
Well Daniel and Dave noted it as well, so I'm just repeating it: Your design choices are not an argument to get something upstream.
It's the job of the maintainers and at the end of the Linus to judge of something is acceptable or not.
As far as I can see a good part of this this idea has been exercised lengthy with KFD and it turned out to not be the best approach.
So from what I've seen the design you outlined is extremely unlikely to go upstream.
Regards,
Christian.
Am 27.01.24 um 03:21 schrieb Zeng, Oak:
Regarding the idea of expanding userptr to support migration, we explored this idea long time ago. It provides similar functions of the system allocator but its interface is not as convenient as system allocator. Besides the shared virtual address space, another benefit of a system allocator is, you can offload cpu program to gpu easier, you don’t need to call driver specific API (such as register_userptr and vm_bind in this case) for memory allocation.
We also scoped the implementation. It turned out to be big, and not as beautiful as hmm. Why we gave up this approach.
From: Christian König <christian.koenig@amd.com><mailto:christian.koenig@amd.com>
Sent: Friday, January 26, 2024 7:52 AM
To: Thomas Hellström <thomas.hellstrom@linux.intel.com><mailto:thomas.hellstrom@linux.intel.com>; Daniel Vetter <daniel@ffwll.ch><mailto:daniel@ffwll.ch>
Cc: Brost, Matthew <matthew.brost@intel.com><mailto:matthew.brost@intel.com>; Felix Kuehling <felix.kuehling@amd.com><mailto:felix.kuehling@amd.com>; Welty, Brian <brian.welty@intel.com><mailto:brian.welty@intel.com>; Ghimiray, Himal Prasad <himal.prasad.ghimiray@intel.com><mailto:himal.prasad.ghimiray@intel.com>; Zeng, Oak <oak.zeng@intel.com><mailto:oak.zeng@intel.com>; Gupta, saurabhg <saurabhg.gupta@intel.com><mailto:saurabhg.gupta@intel.com>; Danilo Krummrich <dakr@redhat.com><mailto:dakr@redhat.com>; dri-devel@lists.freedesktop.org<mailto:dri-devel@lists.freedesktop.org>; Bommu, Krishnaiah <krishnaiah.bommu@intel.com><mailto:krishnaiah.bommu@intel.com>; Dave Airlie <airlied@redhat.com><mailto:airlied@redhat.com>; Vishwanathapura, Niranjana <niranjana.vishwanathapura@intel.com><mailto:niranjana.vishwanathapura@intel.com>; intel-xe@lists.freedesktop.org<mailto:intel-xe@lists.freedesktop.org>
Subject: Re: Making drm_gpuvm work across gpu devices
Am 26.01.24 um 09:21 schrieb Thomas Hellström:
Hi, all
On Thu, 2024-01-25 at 19:32 +0100, Daniel Vetter wrote:
On Wed, Jan 24, 2024 at 09:33:12AM +0100, Christian König wrote:
Am 23.01.24 um 20:37 schrieb Zeng, Oak:
[SNIP]
Yes most API are per device based.
One exception I know is actually the kfd SVM API. If you look at
the svm_ioctl function, it is per-process based. Each kfd_process
represent a process across N gpu devices.
Yeah and that was a big mistake in my opinion. We should really not
do that
ever again.
Need to say, kfd SVM represent a shared virtual address space
across CPU and all GPU devices on the system. This is by the
definition of SVM (shared virtual memory). This is very different
from our legacy gpu *device* driver which works for only one
device (i.e., if you want one device to access another device's
memory, you will have to use dma-buf export/import etc).
Exactly that thinking is what we have currently found as blocker
for a
virtualization projects. Having SVM as device independent feature
which
somehow ties to the process address space turned out to be an
extremely bad
idea.
The background is that this only works for some use cases but not
all of
them.
What's working much better is to just have a mirror functionality
which says
that a range A..B of the process address space is mapped into a
range C..D
of the GPU address space.
Those ranges can then be used to implement the SVM feature required
for
higher level APIs and not something you need at the UAPI or even
inside the
low level kernel memory management.
When you talk about migrating memory to a device you also do this
on a per
device basis and *not* tied to the process address space. If you
then get
crappy performance because userspace gave contradicting information
where to
migrate memory then that's a bug in userspace and not something the
kernel
should try to prevent somehow.
[SNIP]
I think if you start using the same drm_gpuvm for multiple
devices you
will sooner or later start to run into the same mess we have
seen with
KFD, where we moved more and more functionality from the KFD to
the DRM
render node because we found that a lot of the stuff simply
doesn't work
correctly with a single object to maintain the state.
As I understand it, KFD is designed to work across devices. A
single pseudo /dev/kfd device represent all hardware gpu devices.
That is why during kfd open, many pdd (process device data) is
created, each for one hardware device for this process.
Yes, I'm perfectly aware of that. And I can only repeat myself that
I see
this design as a rather extreme failure. And I think it's one of
the reasons
why NVidia is so dominant with Cuda.
This whole approach KFD takes was designed with the idea of
extending the
CPU process into the GPUs, but this idea only works for a few use
cases and
is not something we should apply to drivers in general.
A very good example are virtualization use cases where you end up
with CPU
address != GPU address because the VAs are actually coming from the
guest VM
and not the host process.
SVM is a high level concept of OpenCL, Cuda, ROCm etc.. This should
not have
any influence on the design of the kernel UAPI.
If you want to do something similar as KFD for Xe I think you need
to get
explicit permission to do this from Dave and Daniel and maybe even
Linus.
I think the one and only one exception where an SVM uapi like in kfd
makes
sense, is if the _hardware_ itself, not the software stack defined
semantics that you've happened to build on top of that hw, enforces a
1:1
mapping with the cpu process address space.
Which means your hardware is using PASID, IOMMU based translation,
PCI-ATS
(address translation services) or whatever your hw calls it and has
_no_
device-side pagetables on top. Which from what I've seen all devices
with
device-memory have, simply because they need some place to store
whether
that memory is currently in device memory or should be translated
using
PASID. Currently there's no gpu that works with PASID only, but there
are
some on-cpu-die accelerator things that do work like that.
Maybe in the future there will be some accelerators that are fully
cpu
cache coherent (including atomics) with something like CXL, and the
on-device memory is managed as normal system memory with struct page
as
ZONE_DEVICE and accelerator va -> physical address translation is
only
done with PASID ... but for now I haven't seen that, definitely not
in
upstream drivers.
And the moment you have some per-device pagetables or per-device
memory
management of some sort (like using gpuva mgr) then I'm 100% agreeing
with
Christian that the kfd SVM model is too strict and not a great idea.
Cheers, Sima
I'm trying to digest all the comments here, The end goal is to be able
to support something similar to this here:
https://developer.nvidia.com/blog/simplifying-gpu-application-development-with-heterogeneous-memory-management/
Christian, If I understand you correctly, you're strongly suggesting
not to try to manage a common virtual address space across different
devices in the kernel, but merely providing building blocks to do so,
like for example a generalized userptr with migration support using
HMM; That way each "mirror" of the CPU mm would be per device and
inserted into the gpu_vm just like any other gpu_vma, and user-space
would dictate the A..B -> C..D mapping by choosing the GPU_VA for the
vma.
Exactly that, yes.
Sima, it sounds like you're suggesting to shy away from hmm and not
even attempt to support this except if it can be done using IOMMU sva
on selected hardware?
I think that comment goes more into the direction of: If you have ATS/ATC/PRI capable hardware which exposes the functionality to make memory reads and writes directly into the address space of the CPU then yes an SVM only interface is ok because the hardware can't do anything else. But as long as you have something like GPUVM then please don't restrict yourself.
Which I totally agree on as well. The ATS/ATC/PRI combination doesn't allow using separate page tables device and CPU and so also not separate VAs.
This was one of the reasons why we stopped using this approach for AMD GPUs.
Regards,
Christian.
Could you clarify a bit?
Thanks,
Thomas
[-- Attachment #2: Type: text/html, Size: 30510 bytes --]
WARNING: multiple messages have this Message-ID (diff)
From: "Zeng, Oak" <oak.zeng@intel.com>
To: "Christian König" <christian.koenig@amd.com>,
"Thomas Hellström" <thomas.hellstrom@linux.intel.com>,
"Daniel Vetter" <daniel@ffwll.ch>,
"Dave Airlie" <airlied@redhat.com>
Cc: "rcampbell@nvidia.com" <rcampbell@nvidia.com>,
"apopple@nvidia.com" <apopple@nvidia.com>,
Felix Kuehling <felix.kuehling@amd.com>,
"Shah, Ankur N" <ankur.n.shah@intel.com>,
"dri-devel@lists.freedesktop.org"
<dri-devel@lists.freedesktop.org>,
"jglisse@redhat.com" <jglisse@redhat.com>,
"intel-xe@lists.freedesktop.org" <intel-xe@lists.freedesktop.org>,
Danilo Krummrich <dakr@redhat.com>
Subject: RE: Making drm_gpuvm work across gpu devices
Date: Tue, 30 Jan 2024 22:29:08 +0000 [thread overview]
Message-ID: <SA1PR11MB6991E4CF9949916C6290C232927D2@SA1PR11MB6991.namprd11.prod.outlook.com> (raw)
In-Reply-To: <801c3357-d49b-4505-a1a4-ef2f36ba63e0@amd.com>
[-- Attachment #1: Type: text/plain, Size: 17584 bytes --]
Hi Christian,
Nvidia Nouveau driver uses exactly the same concept of SVM with HMM, GPU address in the same process is exactly the same with CPU virtual address. It is already in upstream Linux kernel. We Intel just follow the same direction for our customers. Why we are not allowed?
From: Christian König <christian.koenig@amd.com>
Sent: Tuesday, January 30, 2024 3:40 AM
To: Zeng, Oak <oak.zeng@intel.com>; Thomas Hellström <thomas.hellstrom@linux.intel.com>; Daniel Vetter <daniel@ffwll.ch>; Dave Airlie <airlied@redhat.com>
Cc: Brost, Matthew <matthew.brost@intel.com>; Felix Kuehling <felix.kuehling@amd.com>; Welty, Brian <brian.welty@intel.com>; dri-devel@lists.freedesktop.org; Ghimiray, Himal Prasad <himal.prasad.ghimiray@intel.com>; Bommu, Krishnaiah <krishnaiah.bommu@intel.com>; Gupta, saurabhg <saurabhg.gupta@intel.com>; Vishwanathapura, Niranjana <niranjana.vishwanathapura@intel.com>; intel-xe@lists.freedesktop.org; Danilo Krummrich <dakr@redhat.com>
Subject: Re: Making drm_gpuvm work across gpu devices
Am 30.01.24 um 01:21 schrieb Zeng, Oak:
The example you used to prove that KFD is a design failure, is against *any* design which utilize system allocator and hmm. The way that one proxy process running on host to handle many guest processes, doesn’t fit into the concept of “share address space b/t cpu and gpu”. The shared address space has to be within one process. Your proxy process represent many guest processes. It is a fundamental conflict.
Also your userptr proposal does’t solve this problem either:
Imagine you have guest process1 mapping CPU address range A…B to GPU address range C…D
And you have guest process 2 mapping CPU address range A…B to GPU address range C…D, since process 1 and 2 are two different process, it is legal for process 2 to do the exact same mapping.
Now when gpu shader access address C…D, a gpu page fault happens, what does your proxy process do? Which guest process will this fault be directed to and handled? Except you have extra information/API to tell proxy process and GPU HW, there is no way to figure out.
Well yes, as far as I can see the fundamental design issue in the KFD is that it ties together CPU and GPU address space. That came from the implementation using the ATS/PRI feature to access the CPU address space from the GPU.
If you don't do ATS/PRI then you don't have that restriction and you can do as many GPU address spaces per CPU process as you want. This is just how the hw works.
So in your example above when you have multiple mappings for the range A..B you also have multiple GPU address spaces and so can distinct where the page fault is coming from just by looking at the source of it. All you then need is userfaultfd() to forward the fault to the client and you are pretty much done.
Compared to the shared virtual address space concept of HMM, the userptr design is nothing new except it allows CPU and GPU to use different address to access the same object. If you replace above C…D with A…B, above description becomes a description of the “problem” of HMM/shared virtual address design.
Both design has the same difficulty with your example of the special virtualization environment setup.
As said, we spent effort scoped the userptr solution some time ago. The problem we found enabling userptr with migration were:
1. The user interface of userptr is not as convenient as system allocator. With the userptr solution, user need to call userptr_ioctl and vm_bind for *every* single cpu pointer that he want to use in a gpu program. While with system allocator, programmer just use any cpu pointer directly in gpu program without any extra driver ioctls.
And I think exactly that is questionable. Why not at least call it for the whole address space once during initialization?
> We don’t see the real benefit of using a different Gpu address C…D than the A..B, except you can prove my above reasoning is wrong. In most use cases, you can make GPU C…D == CPU A…B, why bother then?
Because there are cases where this isn't true. We just recently ran into exactly that use case with a customer. It might be that you will never need this, but again the approach should generally be that the kernel exposes the hardware features and as far as I can see the hardware can do this.
And apart from those use cases there is also another good reason for this: CPU are going towards 5 level of page tables and GPUs are lacking behind. It's not unrealistic to run into cases where you can only mirror parts of the CPU address space into the GPU address space because of hardware restrictions. And in this case you absolutely do want the flexibility to have different address ranges.
> Looked into implementation details, since hmm fundamentally assume a shared virtual address space b/t cpu and device, for the userptr solution to leverage hmm, you need perform address space conversion every time you calls into hmm functions.
Correct, but that is trivial. I mean we do nothing else with VMAs mapping into the address space of files on the CPU either.
Which is by the way a good analogy. The CPU address space consists of anonymous memory and file mappings, where the later covers both real files on a file system as well as devices.
The struct address_space in the Linux kernel for example describes a file address space and not the CPU address space because the later is just a technical tool to form an execution environment which can access the former.
With GPUs it's pretty much the same. You have mappings which can be backed by CPU address space using functionalities like HMM as well as buffer objects created directly through device drivers.
In summary, GPU device is just a piece of HW to accelerate your CPU program.
Well exactly that's not how I see it. CPU accelerators are extensions like SSE, AVX, FPUs etc... GPU are accelerators attached as I/O devices.
And that GPUs are separate to the CPU is a benefit which gives them advantage over CPU based acceleration approaches.
This obviously makes GPUs harder to program and SVM is a method to counter this, but that doesn't make SVM a good design pattern for kernel or device driver interfaces.
If HW allows, it is more convenient to use shared address space b/t cpu and GPU. On old HW (example, no gpu page fault support, or gpu only has a very limited address space), we can disable system allocator/SVM. If you use different address space on modern GPU, why don’t you use different address space on different CPU cores?
Quite simple, modern CPU are homogeneous. From the application point of view they still look more or less the same they did 40 years ago.
GPUs on the other hand look quite a bit different. SVM is now a tool to reduce this difference but it doesn't make the differences in execution environment go away.
And I can only repeat myself that this is actually a good thing, cause otherwise GPUs would loose some of their advantage over CPUs.
Regards,
Christian.
Regards,
Oak
From: dri-devel <dri-devel-bounces@lists.freedesktop.org><mailto:dri-devel-bounces@lists.freedesktop.org> On Behalf Of Christian König
Sent: Monday, January 29, 2024 5:20 AM
To: Zeng, Oak <oak.zeng@intel.com><mailto:oak.zeng@intel.com>; Thomas Hellström <thomas.hellstrom@linux.intel.com><mailto:thomas.hellstrom@linux.intel.com>; Daniel Vetter <daniel@ffwll.ch><mailto:daniel@ffwll.ch>; Dave Airlie <airlied@redhat.com><mailto:airlied@redhat.com>
Cc: Brost, Matthew <matthew.brost@intel.com><mailto:matthew.brost@intel.com>; Felix Kuehling <felix.kuehling@amd.com><mailto:felix.kuehling@amd.com>; Welty, Brian <brian.welty@intel.com><mailto:brian.welty@intel.com>; dri-devel@lists.freedesktop.org<mailto:dri-devel@lists.freedesktop.org>; Ghimiray, Himal Prasad <himal.prasad.ghimiray@intel.com><mailto:himal.prasad.ghimiray@intel.com>; Bommu, Krishnaiah <krishnaiah.bommu@intel.com><mailto:krishnaiah.bommu@intel.com>; Gupta, saurabhg <saurabhg.gupta@intel.com><mailto:saurabhg.gupta@intel.com>; Vishwanathapura, Niranjana <niranjana.vishwanathapura@intel.com><mailto:niranjana.vishwanathapura@intel.com>; intel-xe@lists.freedesktop.org<mailto:intel-xe@lists.freedesktop.org>; Danilo Krummrich <dakr@redhat.com><mailto:dakr@redhat.com>
Subject: Re: Making drm_gpuvm work across gpu devices
Well Daniel and Dave noted it as well, so I'm just repeating it: Your design choices are not an argument to get something upstream.
It's the job of the maintainers and at the end of the Linus to judge of something is acceptable or not.
As far as I can see a good part of this this idea has been exercised lengthy with KFD and it turned out to not be the best approach.
So from what I've seen the design you outlined is extremely unlikely to go upstream.
Regards,
Christian.
Am 27.01.24 um 03:21 schrieb Zeng, Oak:
Regarding the idea of expanding userptr to support migration, we explored this idea long time ago. It provides similar functions of the system allocator but its interface is not as convenient as system allocator. Besides the shared virtual address space, another benefit of a system allocator is, you can offload cpu program to gpu easier, you don’t need to call driver specific API (such as register_userptr and vm_bind in this case) for memory allocation.
We also scoped the implementation. It turned out to be big, and not as beautiful as hmm. Why we gave up this approach.
From: Christian König <christian.koenig@amd.com><mailto:christian.koenig@amd.com>
Sent: Friday, January 26, 2024 7:52 AM
To: Thomas Hellström <thomas.hellstrom@linux.intel.com><mailto:thomas.hellstrom@linux.intel.com>; Daniel Vetter <daniel@ffwll.ch><mailto:daniel@ffwll.ch>
Cc: Brost, Matthew <matthew.brost@intel.com><mailto:matthew.brost@intel.com>; Felix Kuehling <felix.kuehling@amd.com><mailto:felix.kuehling@amd.com>; Welty, Brian <brian.welty@intel.com><mailto:brian.welty@intel.com>; Ghimiray, Himal Prasad <himal.prasad.ghimiray@intel.com><mailto:himal.prasad.ghimiray@intel.com>; Zeng, Oak <oak.zeng@intel.com><mailto:oak.zeng@intel.com>; Gupta, saurabhg <saurabhg.gupta@intel.com><mailto:saurabhg.gupta@intel.com>; Danilo Krummrich <dakr@redhat.com><mailto:dakr@redhat.com>; dri-devel@lists.freedesktop.org<mailto:dri-devel@lists.freedesktop.org>; Bommu, Krishnaiah <krishnaiah.bommu@intel.com><mailto:krishnaiah.bommu@intel.com>; Dave Airlie <airlied@redhat.com><mailto:airlied@redhat.com>; Vishwanathapura, Niranjana <niranjana.vishwanathapura@intel.com><mailto:niranjana.vishwanathapura@intel.com>; intel-xe@lists.freedesktop.org<mailto:intel-xe@lists.freedesktop.org>
Subject: Re: Making drm_gpuvm work across gpu devices
Am 26.01.24 um 09:21 schrieb Thomas Hellström:
Hi, all
On Thu, 2024-01-25 at 19:32 +0100, Daniel Vetter wrote:
On Wed, Jan 24, 2024 at 09:33:12AM +0100, Christian König wrote:
Am 23.01.24 um 20:37 schrieb Zeng, Oak:
[SNIP]
Yes most API are per device based.
One exception I know is actually the kfd SVM API. If you look at
the svm_ioctl function, it is per-process based. Each kfd_process
represent a process across N gpu devices.
Yeah and that was a big mistake in my opinion. We should really not
do that
ever again.
Need to say, kfd SVM represent a shared virtual address space
across CPU and all GPU devices on the system. This is by the
definition of SVM (shared virtual memory). This is very different
from our legacy gpu *device* driver which works for only one
device (i.e., if you want one device to access another device's
memory, you will have to use dma-buf export/import etc).
Exactly that thinking is what we have currently found as blocker
for a
virtualization projects. Having SVM as device independent feature
which
somehow ties to the process address space turned out to be an
extremely bad
idea.
The background is that this only works for some use cases but not
all of
them.
What's working much better is to just have a mirror functionality
which says
that a range A..B of the process address space is mapped into a
range C..D
of the GPU address space.
Those ranges can then be used to implement the SVM feature required
for
higher level APIs and not something you need at the UAPI or even
inside the
low level kernel memory management.
When you talk about migrating memory to a device you also do this
on a per
device basis and *not* tied to the process address space. If you
then get
crappy performance because userspace gave contradicting information
where to
migrate memory then that's a bug in userspace and not something the
kernel
should try to prevent somehow.
[SNIP]
I think if you start using the same drm_gpuvm for multiple
devices you
will sooner or later start to run into the same mess we have
seen with
KFD, where we moved more and more functionality from the KFD to
the DRM
render node because we found that a lot of the stuff simply
doesn't work
correctly with a single object to maintain the state.
As I understand it, KFD is designed to work across devices. A
single pseudo /dev/kfd device represent all hardware gpu devices.
That is why during kfd open, many pdd (process device data) is
created, each for one hardware device for this process.
Yes, I'm perfectly aware of that. And I can only repeat myself that
I see
this design as a rather extreme failure. And I think it's one of
the reasons
why NVidia is so dominant with Cuda.
This whole approach KFD takes was designed with the idea of
extending the
CPU process into the GPUs, but this idea only works for a few use
cases and
is not something we should apply to drivers in general.
A very good example are virtualization use cases where you end up
with CPU
address != GPU address because the VAs are actually coming from the
guest VM
and not the host process.
SVM is a high level concept of OpenCL, Cuda, ROCm etc.. This should
not have
any influence on the design of the kernel UAPI.
If you want to do something similar as KFD for Xe I think you need
to get
explicit permission to do this from Dave and Daniel and maybe even
Linus.
I think the one and only one exception where an SVM uapi like in kfd
makes
sense, is if the _hardware_ itself, not the software stack defined
semantics that you've happened to build on top of that hw, enforces a
1:1
mapping with the cpu process address space.
Which means your hardware is using PASID, IOMMU based translation,
PCI-ATS
(address translation services) or whatever your hw calls it and has
_no_
device-side pagetables on top. Which from what I've seen all devices
with
device-memory have, simply because they need some place to store
whether
that memory is currently in device memory or should be translated
using
PASID. Currently there's no gpu that works with PASID only, but there
are
some on-cpu-die accelerator things that do work like that.
Maybe in the future there will be some accelerators that are fully
cpu
cache coherent (including atomics) with something like CXL, and the
on-device memory is managed as normal system memory with struct page
as
ZONE_DEVICE and accelerator va -> physical address translation is
only
done with PASID ... but for now I haven't seen that, definitely not
in
upstream drivers.
And the moment you have some per-device pagetables or per-device
memory
management of some sort (like using gpuva mgr) then I'm 100% agreeing
with
Christian that the kfd SVM model is too strict and not a great idea.
Cheers, Sima
I'm trying to digest all the comments here, The end goal is to be able
to support something similar to this here:
https://developer.nvidia.com/blog/simplifying-gpu-application-development-with-heterogeneous-memory-management/
Christian, If I understand you correctly, you're strongly suggesting
not to try to manage a common virtual address space across different
devices in the kernel, but merely providing building blocks to do so,
like for example a generalized userptr with migration support using
HMM; That way each "mirror" of the CPU mm would be per device and
inserted into the gpu_vm just like any other gpu_vma, and user-space
would dictate the A..B -> C..D mapping by choosing the GPU_VA for the
vma.
Exactly that, yes.
Sima, it sounds like you're suggesting to shy away from hmm and not
even attempt to support this except if it can be done using IOMMU sva
on selected hardware?
I think that comment goes more into the direction of: If you have ATS/ATC/PRI capable hardware which exposes the functionality to make memory reads and writes directly into the address space of the CPU then yes an SVM only interface is ok because the hardware can't do anything else. But as long as you have something like GPUVM then please don't restrict yourself.
Which I totally agree on as well. The ATS/ATC/PRI combination doesn't allow using separate page tables device and CPU and so also not separate VAs.
This was one of the reasons why we stopped using this approach for AMD GPUs.
Regards,
Christian.
Could you clarify a bit?
Thanks,
Thomas
[-- Attachment #2: Type: text/html, Size: 30510 bytes --]
next prev parent reply other threads:[~2024-01-30 22:29 UTC|newest]
Thread overview: 198+ messages / expand[flat|nested] mbox.gz Atom feed top
2024-01-17 22:12 [PATCH 00/23] XeKmd basic SVM support Oak Zeng
2024-01-17 22:12 ` Oak Zeng
2024-01-17 22:12 ` [PATCH 01/23] drm/xe/svm: Add SVM document Oak Zeng
2024-01-17 22:12 ` Oak Zeng
2024-01-17 22:12 ` [PATCH 02/23] drm/xe/svm: Add svm key data structures Oak Zeng
2024-01-17 22:12 ` Oak Zeng
2024-01-17 22:12 ` [PATCH 03/23] drm/xe/svm: create xe svm during vm creation Oak Zeng
2024-01-17 22:12 ` Oak Zeng
2024-01-17 22:12 ` [PATCH 04/23] drm/xe/svm: Trace svm creation Oak Zeng
2024-01-17 22:12 ` Oak Zeng
2024-01-17 22:12 ` [PATCH 05/23] drm/xe/svm: add helper to retrieve svm range from address Oak Zeng
2024-01-17 22:12 ` Oak Zeng
2024-01-17 22:12 ` [PATCH 06/23] drm/xe/svm: Introduce a helper to build sg table from hmm range Oak Zeng
2024-01-17 22:12 ` Oak Zeng
2024-04-05 0:39 ` Jason Gunthorpe
2024-04-05 3:33 ` Zeng, Oak
2024-04-05 12:37 ` Jason Gunthorpe
2024-04-05 16:42 ` Zeng, Oak
2024-04-05 18:02 ` Jason Gunthorpe
2024-04-09 16:45 ` Zeng, Oak
2024-04-09 17:24 ` Jason Gunthorpe
2024-04-23 21:17 ` Zeng, Oak
2024-04-24 2:31 ` Matthew Brost
2024-04-24 13:57 ` Jason Gunthorpe
2024-04-24 16:35 ` Matthew Brost
2024-04-24 16:44 ` Jason Gunthorpe
2024-04-24 16:56 ` Matthew Brost
2024-04-24 17:48 ` Jason Gunthorpe
2024-04-24 13:48 ` Jason Gunthorpe
2024-04-24 23:59 ` Zeng, Oak
2024-04-25 1:05 ` Jason Gunthorpe
2024-04-26 9:55 ` Thomas Hellström
2024-04-26 12:00 ` Jason Gunthorpe
2024-04-26 14:49 ` Thomas Hellström
2024-04-26 16:35 ` Jason Gunthorpe
2024-04-29 8:25 ` Thomas Hellström
2024-04-30 17:30 ` Jason Gunthorpe
2024-04-30 18:57 ` Daniel Vetter
2024-05-01 0:09 ` Jason Gunthorpe
2024-05-02 8:04 ` Daniel Vetter
2024-05-02 9:11 ` Thomas Hellström
2024-05-02 12:46 ` Jason Gunthorpe
2024-05-02 15:01 ` Thomas Hellström
2024-05-02 19:25 ` Zeng, Oak
2024-05-03 13:37 ` Jason Gunthorpe
2024-05-03 14:43 ` Zeng, Oak
2024-05-03 16:28 ` Jason Gunthorpe
2024-05-03 20:29 ` Zeng, Oak
2024-05-04 1:03 ` Dave Airlie
2024-05-06 13:04 ` Daniel Vetter
2024-05-06 23:50 ` Matthew Brost
2024-05-07 11:56 ` Jason Gunthorpe
2024-05-06 13:33 ` Jason Gunthorpe
2024-04-09 17:33 ` Matthew Brost
2024-01-17 22:12 ` [PATCH 07/23] drm/xe/svm: Add helper for binding hmm range to gpu Oak Zeng
2024-01-17 22:12 ` Oak Zeng
2024-01-17 22:12 ` [PATCH 08/23] drm/xe/svm: Add helper to invalidate svm range from GPU Oak Zeng
2024-01-17 22:12 ` Oak Zeng
2024-01-17 22:12 ` [PATCH 09/23] drm/xe/svm: Remap and provide memmap backing for GPU vram Oak Zeng
2024-01-17 22:12 ` Oak Zeng
2024-01-17 22:12 ` [PATCH 10/23] drm/xe/svm: Introduce svm migration function Oak Zeng
2024-01-17 22:12 ` Oak Zeng
2024-01-17 22:12 ` [PATCH 11/23] drm/xe/svm: implement functions to allocate and free device memory Oak Zeng
2024-01-17 22:12 ` Oak Zeng
2024-01-17 22:12 ` [PATCH 12/23] drm/xe/svm: Trace buddy block allocation and free Oak Zeng
2024-01-17 22:12 ` Oak Zeng
2024-01-17 22:12 ` [PATCH 13/23] drm/xe/svm: Handle CPU page fault Oak Zeng
2024-01-17 22:12 ` Oak Zeng
2024-01-17 22:12 ` [PATCH 14/23] drm/xe/svm: trace svm range migration Oak Zeng
2024-01-17 22:12 ` Oak Zeng
2024-01-17 22:12 ` [PATCH 15/23] drm/xe/svm: Implement functions to register and unregister mmu notifier Oak Zeng
2024-01-17 22:12 ` Oak Zeng
2024-01-17 22:12 ` [PATCH 16/23] drm/xe/svm: Implement the mmu notifier range invalidate callback Oak Zeng
2024-01-17 22:12 ` Oak Zeng
2024-01-17 22:12 ` [PATCH 17/23] drm/xe/svm: clean up svm range during process exit Oak Zeng
2024-01-17 22:12 ` Oak Zeng
2024-01-17 22:12 ` [PATCH 18/23] drm/xe/svm: Move a few structures to xe_gt.h Oak Zeng
2024-01-17 22:12 ` Oak Zeng
2024-01-17 22:12 ` [PATCH 19/23] drm/xe/svm: migrate svm range to vram Oak Zeng
2024-01-17 22:12 ` Oak Zeng
2024-01-17 22:12 ` [PATCH 20/23] drm/xe/svm: Populate svm range Oak Zeng
2024-01-17 22:12 ` Oak Zeng
2024-01-17 22:12 ` [PATCH 21/23] drm/xe/svm: GPU page fault support Oak Zeng
2024-01-17 22:12 ` Oak Zeng
2024-01-23 2:06 ` Welty, Brian
2024-01-23 2:06 ` Welty, Brian
2024-01-23 3:09 ` Zeng, Oak
2024-01-23 3:09 ` Zeng, Oak
2024-01-23 3:21 ` Making drm_gpuvm work across gpu devices Zeng, Oak
2024-01-23 3:21 ` Zeng, Oak
2024-01-23 11:13 ` Christian König
2024-01-23 11:13 ` Christian König
2024-01-23 19:37 ` Zeng, Oak
2024-01-23 19:37 ` Zeng, Oak
2024-01-23 20:17 ` Felix Kuehling
2024-01-23 20:17 ` Felix Kuehling
2024-01-25 1:39 ` Zeng, Oak
2024-01-25 1:39 ` Zeng, Oak
2024-01-23 23:56 ` Danilo Krummrich
2024-01-23 23:56 ` Danilo Krummrich
2024-01-24 3:57 ` Zeng, Oak
2024-01-24 3:57 ` Zeng, Oak
2024-01-24 4:14 ` Zeng, Oak
2024-01-24 4:14 ` Zeng, Oak
2024-01-24 6:48 ` Christian König
2024-01-24 6:48 ` Christian König
2024-01-25 22:13 ` Danilo Krummrich
2024-01-25 22:13 ` Danilo Krummrich
2024-01-24 8:33 ` Christian König
2024-01-24 8:33 ` Christian König
2024-01-25 1:17 ` Zeng, Oak
2024-01-25 1:17 ` Zeng, Oak
2024-01-25 1:25 ` David Airlie
2024-01-25 1:25 ` David Airlie
2024-01-25 5:25 ` Zeng, Oak
2024-01-25 5:25 ` Zeng, Oak
2024-01-26 10:09 ` Christian König
2024-01-26 10:09 ` Christian König
2024-01-26 20:13 ` Zeng, Oak
2024-01-26 20:13 ` Zeng, Oak
2024-01-29 10:10 ` Christian König
2024-01-29 10:10 ` Christian König
2024-01-29 20:09 ` Zeng, Oak
2024-01-29 20:09 ` Zeng, Oak
2024-01-25 11:00 ` 回复：Making " 周春明(日月)
2024-01-25 11:00 ` 周春明(日月)
2024-01-25 17:00 ` Zeng, Oak
2024-01-25 17:00 ` Zeng, Oak
2024-01-25 17:15 ` Making " Felix Kuehling
2024-01-25 17:15 ` Felix Kuehling
2024-01-25 18:37 ` Zeng, Oak
2024-01-25 18:37 ` Zeng, Oak
2024-01-26 13:23 ` Christian König
2024-01-26 13:23 ` Christian König
2024-01-25 16:42 ` Zeng, Oak
2024-01-25 16:42 ` Zeng, Oak
2024-01-25 18:32 ` Daniel Vetter
2024-01-25 18:32 ` Daniel Vetter
2024-01-25 21:02 ` Zeng, Oak
2024-01-25 21:02 ` Zeng, Oak
2024-01-26 8:21 ` Thomas Hellström
2024-01-26 8:21 ` Thomas Hellström
2024-01-26 12:52 ` Christian König
2024-01-26 12:52 ` Christian König
2024-01-27 2:21 ` Zeng, Oak
2024-01-27 2:21 ` Zeng, Oak
2024-01-29 10:19 ` Christian König
2024-01-29 10:19 ` Christian König
2024-01-30 0:21 ` Zeng, Oak
2024-01-30 0:21 ` Zeng, Oak
2024-01-30 8:39 ` Christian König
2024-01-30 8:39 ` Christian König
2024-01-30 22:29 ` Zeng, Oak [this message]
2024-01-30 22:29 ` Zeng, Oak
2024-01-30 23:12 ` David Airlie
2024-01-30 23:12 ` David Airlie
2024-01-31 9:15 ` Daniel Vetter
2024-01-31 9:15 ` Daniel Vetter
2024-01-31 20:17 ` Zeng, Oak
2024-01-31 20:17 ` Zeng, Oak
2024-01-31 20:59 ` Zeng, Oak
2024-01-31 20:59 ` Zeng, Oak
2024-02-01 8:52 ` Christian König
2024-02-01 8:52 ` Christian König
2024-02-29 18:22 ` Zeng, Oak
2024-03-08 4:43 ` Zeng, Oak
2024-03-08 10:07 ` Christian König
2024-01-30 8:43 ` Thomas Hellström
2024-01-30 8:43 ` Thomas Hellström
2024-01-29 15:03 ` Felix Kuehling
2024-01-29 15:03 ` Felix Kuehling
2024-01-29 15:33 ` Christian König
2024-01-29 15:33 ` Christian König
2024-01-29 16:24 ` Felix Kuehling
2024-01-29 16:24 ` Felix Kuehling
2024-01-29 16:28 ` Christian König
2024-01-29 16:28 ` Christian König
2024-01-29 17:52 ` Felix Kuehling
2024-01-29 17:52 ` Felix Kuehling
2024-01-29 19:03 ` Christian König
2024-01-29 19:03 ` Christian König
2024-01-29 20:24 ` Felix Kuehling
2024-01-29 20:24 ` Felix Kuehling
2024-02-23 20:12 ` Zeng, Oak
2024-02-27 6:54 ` Christian König
2024-02-27 15:58 ` Zeng, Oak
2024-02-28 19:51 ` Zeng, Oak
2024-02-29 9:41 ` Christian König
2024-02-29 16:05 ` Zeng, Oak
2024-02-29 17:12 ` Thomas Hellström
2024-03-01 7:01 ` Christian König
2024-01-17 22:12 ` [PATCH 22/23] drm/xe/svm: Add DRM_XE_SVM kernel config entry Oak Zeng
2024-01-17 22:12 ` Oak Zeng
2024-01-17 22:12 ` [PATCH 23/23] drm/xe/svm: Add svm memory hints interface Oak Zeng
2024-01-17 22:12 ` Oak Zeng
2024-01-18 2:45 ` ✓ CI.Patch_applied: success for XeKmd basic SVM support Patchwork
2024-01-18 2:46 ` ✗ CI.checkpatch: warning " Patchwork
2024-01-18 2:46 ` ✗ CI.KUnit: failure " Patchwork
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=SA1PR11MB6991E4CF9949916C6290C232927D2@SA1PR11MB6991.namprd11.prod.outlook.com \
--to=oak.zeng@intel.com \
--cc=airlied@redhat.com \
--cc=ankur.n.shah@intel.com \
--cc=apopple@nvidia.com \
--cc=brian.welty@intel.com \
--cc=christian.koenig@amd.com \
--cc=dakr@redhat.com \
--cc=daniel@ffwll.ch \
--cc=dri-devel@lists.freedesktop.org \
--cc=felix.kuehling@amd.com \
--cc=himal.prasad.ghimiray@intel.com \
--cc=intel-xe@lists.freedesktop.org \
--cc=jglisse@redhat.com \
--cc=krishnaiah.bommu@intel.com \
--cc=matthew.brost@intel.com \
--cc=niranjana.vishwanathapura@intel.com \
--cc=rcampbell@nvidia.com \
--cc=saurabhg.gupta@intel.com \
--cc=thomas.hellstrom@linux.intel.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.