From: Oak Zeng <oak.zeng@intel.com> To: dri-devel@lists.freedesktop.org, intel-xe@lists.freedesktop.org Cc: matthew.brost@intel.com, Thomas.Hellstrom@linux.intel.com, brian.welty@intel.com, himal.prasad.ghimiray@intel.com, krishnaiah.bommu@intel.com, niranjana.vishwanathapura@intel.com Subject: [PATCH 16/23] drm/xe/svm: Implement the mmu notifier range invalidate callback Date: Wed, 17 Jan 2024 17:12:16 -0500 [thread overview] Message-ID: <20240117221223.18540-17-oak.zeng@intel.com> (raw) In-Reply-To: <20240117221223.18540-1-oak.zeng@intel.com> To mirror the CPU page table from GPU side, we register a mmu interval notifier (in the coming patch of this series). Core mm call back to GPU driver whenever there is a change to certain virtual address range, i.e., range is released or unmapped by user etc. This patch implemented the GPU driver callback function for such mmu interval notifier. In the callback function we unbind the address range from GPU if it is unmapped from CPU side, thus we mirror the CPU page table change. We also unregister the mmu interval notifier from core mm in the case of munmap event. But we can't unregister mmu notifier directly from the mmu notifier range invalidation callback function. The reason is, during a munmap (see kernel function vm_munmap), a mmap_write_lock is held, but unregister mmu notifier (calling mmu_interval_notifier_remove) also requires a mmap_write_lock of the current process. Thus, we start a kernel worker to unregister mmu interval notifier on a MMU_NOTIFY_UNMAP event. Signed-off-by: Oak Zeng <oak.zeng@intel.com> Co-developed-by: Niranjana Vishwanathapura <niranjana.vishwanathapura@intel.com> Signed-off-by: Niranjana Vishwanathapura <niranjana.vishwanathapura@intel.com> Cc: Matthew Brost <matthew.brost@intel.com> Cc: Thomas Hellström <thomas.hellstrom@intel.com> Cc: Brian Welty <brian.welty@intel.com> --- drivers/gpu/drm/xe/xe_svm.c | 1 + drivers/gpu/drm/xe/xe_svm.h | 1 - drivers/gpu/drm/xe/xe_svm_range.c | 37 ++++++++++++++++++++++++++++++- 3 files changed, 37 insertions(+), 2 deletions(-) diff --git a/drivers/gpu/drm/xe/xe_svm.c b/drivers/gpu/drm/xe/xe_svm.c index ab3cc2121869..6393251c0051 100644 --- a/drivers/gpu/drm/xe/xe_svm.c +++ b/drivers/gpu/drm/xe/xe_svm.c @@ -8,6 +8,7 @@ #include "xe_svm.h" #include <linux/hmm.h> #include <linux/scatterlist.h> +#include "xe_pt.h" DEFINE_HASHTABLE(xe_svm_table, XE_MAX_SVM_PROCESS); diff --git a/drivers/gpu/drm/xe/xe_svm.h b/drivers/gpu/drm/xe/xe_svm.h index 90e665f2bfc6..0038f98c0cc7 100644 --- a/drivers/gpu/drm/xe/xe_svm.h +++ b/drivers/gpu/drm/xe/xe_svm.h @@ -54,7 +54,6 @@ struct xe_svm { struct xe_svm_range { /** @svm: pointer of the xe_svm that this range belongs to */ struct xe_svm *svm; - /** @notifier: The mmu interval notifer used to keep track of CPU * side address range change. Driver will get a callback with this * notifier if anything changed from CPU side, such as range is diff --git a/drivers/gpu/drm/xe/xe_svm_range.c b/drivers/gpu/drm/xe/xe_svm_range.c index 286d5f7d6ecd..53dd3be7ab9f 100644 --- a/drivers/gpu/drm/xe/xe_svm_range.c +++ b/drivers/gpu/drm/xe/xe_svm_range.c @@ -10,6 +10,7 @@ #include <linux/mutex.h> #include <linux/mm.h> #include "xe_svm.h" +#include "xe_pt.h" /** * xe_svm_range_from_addr() - retrieve svm_range contains a virtual address @@ -59,8 +60,42 @@ bool xe_svm_range_belongs_to_vma(struct mm_struct *mm, return (vma1 == vma) && (vma2 == vma); } +static bool xe_svm_range_invalidate(struct mmu_interval_notifier *mni, + const struct mmu_notifier_range *range, + unsigned long cur_seq) +{ + struct xe_svm_range *svm_range = + container_of(mni, struct xe_svm_range, notifier); + struct xe_svm *svm = svm_range->svm; + unsigned long length = range->end - range->start; + + /* + * MMU_NOTIFY_RELEASE is called upon process exit to notify driver + * to release any process resources, such as zap GPU page table + * mapping or unregister mmu notifier etc. We already clear GPU + * page table and unregister mmu notifier in in xe_destroy_svm, + * upon process exit. So just simply return here. + */ + if (range->event == MMU_NOTIFY_RELEASE) + return true; + + if (mmu_notifier_range_blockable(range)) + mutex_lock(&svm->mutex); + else if (!mutex_trylock(&svm->mutex)) + return false; + + mmu_interval_set_seq(mni, cur_seq); + xe_invalidate_svm_range(svm->vm, range->start, length); + mutex_unlock(&svm->mutex); + + if (range->event == MMU_NOTIFY_UNMAP) + queue_work(system_unbound_wq, &svm_range->unregister_notifier_work); + + return true; +} + static const struct mmu_interval_notifier_ops xe_svm_mni_ops = { - .invalidate = NULL, + .invalidate = xe_svm_range_invalidate, }; /** -- 2.26.3
WARNING: multiple messages have this Message-ID (diff)
From: Oak Zeng <oak.zeng@intel.com> To: dri-devel@lists.freedesktop.org, intel-xe@lists.freedesktop.org Subject: [PATCH 16/23] drm/xe/svm: Implement the mmu notifier range invalidate callback Date: Wed, 17 Jan 2024 17:12:16 -0500 [thread overview] Message-ID: <20240117221223.18540-17-oak.zeng@intel.com> (raw) In-Reply-To: <20240117221223.18540-1-oak.zeng@intel.com> To mirror the CPU page table from GPU side, we register a mmu interval notifier (in the coming patch of this series). Core mm call back to GPU driver whenever there is a change to certain virtual address range, i.e., range is released or unmapped by user etc. This patch implemented the GPU driver callback function for such mmu interval notifier. In the callback function we unbind the address range from GPU if it is unmapped from CPU side, thus we mirror the CPU page table change. We also unregister the mmu interval notifier from core mm in the case of munmap event. But we can't unregister mmu notifier directly from the mmu notifier range invalidation callback function. The reason is, during a munmap (see kernel function vm_munmap), a mmap_write_lock is held, but unregister mmu notifier (calling mmu_interval_notifier_remove) also requires a mmap_write_lock of the current process. Thus, we start a kernel worker to unregister mmu interval notifier on a MMU_NOTIFY_UNMAP event. Signed-off-by: Oak Zeng <oak.zeng@intel.com> Co-developed-by: Niranjana Vishwanathapura <niranjana.vishwanathapura@intel.com> Signed-off-by: Niranjana Vishwanathapura <niranjana.vishwanathapura@intel.com> Cc: Matthew Brost <matthew.brost@intel.com> Cc: Thomas Hellström <thomas.hellstrom@intel.com> Cc: Brian Welty <brian.welty@intel.com> --- drivers/gpu/drm/xe/xe_svm.c | 1 + drivers/gpu/drm/xe/xe_svm.h | 1 - drivers/gpu/drm/xe/xe_svm_range.c | 37 ++++++++++++++++++++++++++++++- 3 files changed, 37 insertions(+), 2 deletions(-) diff --git a/drivers/gpu/drm/xe/xe_svm.c b/drivers/gpu/drm/xe/xe_svm.c index ab3cc2121869..6393251c0051 100644 --- a/drivers/gpu/drm/xe/xe_svm.c +++ b/drivers/gpu/drm/xe/xe_svm.c @@ -8,6 +8,7 @@ #include "xe_svm.h" #include <linux/hmm.h> #include <linux/scatterlist.h> +#include "xe_pt.h" DEFINE_HASHTABLE(xe_svm_table, XE_MAX_SVM_PROCESS); diff --git a/drivers/gpu/drm/xe/xe_svm.h b/drivers/gpu/drm/xe/xe_svm.h index 90e665f2bfc6..0038f98c0cc7 100644 --- a/drivers/gpu/drm/xe/xe_svm.h +++ b/drivers/gpu/drm/xe/xe_svm.h @@ -54,7 +54,6 @@ struct xe_svm { struct xe_svm_range { /** @svm: pointer of the xe_svm that this range belongs to */ struct xe_svm *svm; - /** @notifier: The mmu interval notifer used to keep track of CPU * side address range change. Driver will get a callback with this * notifier if anything changed from CPU side, such as range is diff --git a/drivers/gpu/drm/xe/xe_svm_range.c b/drivers/gpu/drm/xe/xe_svm_range.c index 286d5f7d6ecd..53dd3be7ab9f 100644 --- a/drivers/gpu/drm/xe/xe_svm_range.c +++ b/drivers/gpu/drm/xe/xe_svm_range.c @@ -10,6 +10,7 @@ #include <linux/mutex.h> #include <linux/mm.h> #include "xe_svm.h" +#include "xe_pt.h" /** * xe_svm_range_from_addr() - retrieve svm_range contains a virtual address @@ -59,8 +60,42 @@ bool xe_svm_range_belongs_to_vma(struct mm_struct *mm, return (vma1 == vma) && (vma2 == vma); } +static bool xe_svm_range_invalidate(struct mmu_interval_notifier *mni, + const struct mmu_notifier_range *range, + unsigned long cur_seq) +{ + struct xe_svm_range *svm_range = + container_of(mni, struct xe_svm_range, notifier); + struct xe_svm *svm = svm_range->svm; + unsigned long length = range->end - range->start; + + /* + * MMU_NOTIFY_RELEASE is called upon process exit to notify driver + * to release any process resources, such as zap GPU page table + * mapping or unregister mmu notifier etc. We already clear GPU + * page table and unregister mmu notifier in in xe_destroy_svm, + * upon process exit. So just simply return here. + */ + if (range->event == MMU_NOTIFY_RELEASE) + return true; + + if (mmu_notifier_range_blockable(range)) + mutex_lock(&svm->mutex); + else if (!mutex_trylock(&svm->mutex)) + return false; + + mmu_interval_set_seq(mni, cur_seq); + xe_invalidate_svm_range(svm->vm, range->start, length); + mutex_unlock(&svm->mutex); + + if (range->event == MMU_NOTIFY_UNMAP) + queue_work(system_unbound_wq, &svm_range->unregister_notifier_work); + + return true; +} + static const struct mmu_interval_notifier_ops xe_svm_mni_ops = { - .invalidate = NULL, + .invalidate = xe_svm_range_invalidate, }; /** -- 2.26.3
next prev parent reply other threads:[~2024-01-17 22:02 UTC|newest] Thread overview: 198+ messages / expand[flat|nested] mbox.gz Atom feed top 2024-01-17 22:12 [PATCH 00/23] XeKmd basic SVM support Oak Zeng 2024-01-17 22:12 ` Oak Zeng 2024-01-17 22:12 ` [PATCH 01/23] drm/xe/svm: Add SVM document Oak Zeng 2024-01-17 22:12 ` Oak Zeng 2024-01-17 22:12 ` [PATCH 02/23] drm/xe/svm: Add svm key data structures Oak Zeng 2024-01-17 22:12 ` Oak Zeng 2024-01-17 22:12 ` [PATCH 03/23] drm/xe/svm: create xe svm during vm creation Oak Zeng 2024-01-17 22:12 ` Oak Zeng 2024-01-17 22:12 ` [PATCH 04/23] drm/xe/svm: Trace svm creation Oak Zeng 2024-01-17 22:12 ` Oak Zeng 2024-01-17 22:12 ` [PATCH 05/23] drm/xe/svm: add helper to retrieve svm range from address Oak Zeng 2024-01-17 22:12 ` Oak Zeng 2024-01-17 22:12 ` [PATCH 06/23] drm/xe/svm: Introduce a helper to build sg table from hmm range Oak Zeng 2024-01-17 22:12 ` Oak Zeng 2024-04-05 0:39 ` Jason Gunthorpe 2024-04-05 3:33 ` Zeng, Oak 2024-04-05 12:37 ` Jason Gunthorpe 2024-04-05 16:42 ` Zeng, Oak 2024-04-05 18:02 ` Jason Gunthorpe 2024-04-09 16:45 ` Zeng, Oak 2024-04-09 17:24 ` Jason Gunthorpe 2024-04-23 21:17 ` Zeng, Oak 2024-04-24 2:31 ` Matthew Brost 2024-04-24 13:57 ` Jason Gunthorpe 2024-04-24 16:35 ` Matthew Brost 2024-04-24 16:44 ` Jason Gunthorpe 2024-04-24 16:56 ` Matthew Brost 2024-04-24 17:48 ` Jason Gunthorpe 2024-04-24 13:48 ` Jason Gunthorpe 2024-04-24 23:59 ` Zeng, Oak 2024-04-25 1:05 ` Jason Gunthorpe 2024-04-26 9:55 ` Thomas Hellström 2024-04-26 12:00 ` Jason Gunthorpe 2024-04-26 14:49 ` Thomas Hellström 2024-04-26 16:35 ` Jason Gunthorpe 2024-04-29 8:25 ` Thomas Hellström 2024-04-30 17:30 ` Jason Gunthorpe 2024-04-30 18:57 ` Daniel Vetter 2024-05-01 0:09 ` Jason Gunthorpe 2024-05-02 8:04 ` Daniel Vetter 2024-05-02 9:11 ` Thomas Hellström 2024-05-02 12:46 ` Jason Gunthorpe 2024-05-02 15:01 ` Thomas Hellström 2024-05-02 19:25 ` Zeng, Oak 2024-05-03 13:37 ` Jason Gunthorpe 2024-05-03 14:43 ` Zeng, Oak 2024-05-03 16:28 ` Jason Gunthorpe 2024-05-03 20:29 ` Zeng, Oak 2024-05-04 1:03 ` Dave Airlie 2024-05-06 13:04 ` Daniel Vetter 2024-05-06 23:50 ` Matthew Brost 2024-05-07 11:56 ` Jason Gunthorpe 2024-05-06 13:33 ` Jason Gunthorpe 2024-04-09 17:33 ` Matthew Brost 2024-01-17 22:12 ` [PATCH 07/23] drm/xe/svm: Add helper for binding hmm range to gpu Oak Zeng 2024-01-17 22:12 ` Oak Zeng 2024-01-17 22:12 ` [PATCH 08/23] drm/xe/svm: Add helper to invalidate svm range from GPU Oak Zeng 2024-01-17 22:12 ` Oak Zeng 2024-01-17 22:12 ` [PATCH 09/23] drm/xe/svm: Remap and provide memmap backing for GPU vram Oak Zeng 2024-01-17 22:12 ` Oak Zeng 2024-01-17 22:12 ` [PATCH 10/23] drm/xe/svm: Introduce svm migration function Oak Zeng 2024-01-17 22:12 ` Oak Zeng 2024-01-17 22:12 ` [PATCH 11/23] drm/xe/svm: implement functions to allocate and free device memory Oak Zeng 2024-01-17 22:12 ` Oak Zeng 2024-01-17 22:12 ` [PATCH 12/23] drm/xe/svm: Trace buddy block allocation and free Oak Zeng 2024-01-17 22:12 ` Oak Zeng 2024-01-17 22:12 ` [PATCH 13/23] drm/xe/svm: Handle CPU page fault Oak Zeng 2024-01-17 22:12 ` Oak Zeng 2024-01-17 22:12 ` [PATCH 14/23] drm/xe/svm: trace svm range migration Oak Zeng 2024-01-17 22:12 ` Oak Zeng 2024-01-17 22:12 ` [PATCH 15/23] drm/xe/svm: Implement functions to register and unregister mmu notifier Oak Zeng 2024-01-17 22:12 ` Oak Zeng 2024-01-17 22:12 ` Oak Zeng [this message] 2024-01-17 22:12 ` [PATCH 16/23] drm/xe/svm: Implement the mmu notifier range invalidate callback Oak Zeng 2024-01-17 22:12 ` [PATCH 17/23] drm/xe/svm: clean up svm range during process exit Oak Zeng 2024-01-17 22:12 ` Oak Zeng 2024-01-17 22:12 ` [PATCH 18/23] drm/xe/svm: Move a few structures to xe_gt.h Oak Zeng 2024-01-17 22:12 ` Oak Zeng 2024-01-17 22:12 ` [PATCH 19/23] drm/xe/svm: migrate svm range to vram Oak Zeng 2024-01-17 22:12 ` Oak Zeng 2024-01-17 22:12 ` [PATCH 20/23] drm/xe/svm: Populate svm range Oak Zeng 2024-01-17 22:12 ` Oak Zeng 2024-01-17 22:12 ` [PATCH 21/23] drm/xe/svm: GPU page fault support Oak Zeng 2024-01-17 22:12 ` Oak Zeng 2024-01-23 2:06 ` Welty, Brian 2024-01-23 2:06 ` Welty, Brian 2024-01-23 3:09 ` Zeng, Oak 2024-01-23 3:09 ` Zeng, Oak 2024-01-23 3:21 ` Making drm_gpuvm work across gpu devices Zeng, Oak 2024-01-23 3:21 ` Zeng, Oak 2024-01-23 11:13 ` Christian König 2024-01-23 11:13 ` Christian König 2024-01-23 19:37 ` Zeng, Oak 2024-01-23 19:37 ` Zeng, Oak 2024-01-23 20:17 ` Felix Kuehling 2024-01-23 20:17 ` Felix Kuehling 2024-01-25 1:39 ` Zeng, Oak 2024-01-25 1:39 ` Zeng, Oak 2024-01-23 23:56 ` Danilo Krummrich 2024-01-23 23:56 ` Danilo Krummrich 2024-01-24 3:57 ` Zeng, Oak 2024-01-24 3:57 ` Zeng, Oak 2024-01-24 4:14 ` Zeng, Oak 2024-01-24 4:14 ` Zeng, Oak 2024-01-24 6:48 ` Christian König 2024-01-24 6:48 ` Christian König 2024-01-25 22:13 ` Danilo Krummrich 2024-01-25 22:13 ` Danilo Krummrich 2024-01-24 8:33 ` Christian König 2024-01-24 8:33 ` Christian König 2024-01-25 1:17 ` Zeng, Oak 2024-01-25 1:17 ` Zeng, Oak 2024-01-25 1:25 ` David Airlie 2024-01-25 1:25 ` David Airlie 2024-01-25 5:25 ` Zeng, Oak 2024-01-25 5:25 ` Zeng, Oak 2024-01-26 10:09 ` Christian König 2024-01-26 10:09 ` Christian König 2024-01-26 20:13 ` Zeng, Oak 2024-01-26 20:13 ` Zeng, Oak 2024-01-29 10:10 ` Christian König 2024-01-29 10:10 ` Christian König 2024-01-29 20:09 ` Zeng, Oak 2024-01-29 20:09 ` Zeng, Oak 2024-01-25 11:00 ` 回复:Making " 周春明(日月) 2024-01-25 11:00 ` 周春明(日月) 2024-01-25 17:00 ` Zeng, Oak 2024-01-25 17:00 ` Zeng, Oak 2024-01-25 17:15 ` Making " Felix Kuehling 2024-01-25 17:15 ` Felix Kuehling 2024-01-25 18:37 ` Zeng, Oak 2024-01-25 18:37 ` Zeng, Oak 2024-01-26 13:23 ` Christian König 2024-01-26 13:23 ` Christian König 2024-01-25 16:42 ` Zeng, Oak 2024-01-25 16:42 ` Zeng, Oak 2024-01-25 18:32 ` Daniel Vetter 2024-01-25 18:32 ` Daniel Vetter 2024-01-25 21:02 ` Zeng, Oak 2024-01-25 21:02 ` Zeng, Oak 2024-01-26 8:21 ` Thomas Hellström 2024-01-26 8:21 ` Thomas Hellström 2024-01-26 12:52 ` Christian König 2024-01-26 12:52 ` Christian König 2024-01-27 2:21 ` Zeng, Oak 2024-01-27 2:21 ` Zeng, Oak 2024-01-29 10:19 ` Christian König 2024-01-29 10:19 ` Christian König 2024-01-30 0:21 ` Zeng, Oak 2024-01-30 0:21 ` Zeng, Oak 2024-01-30 8:39 ` Christian König 2024-01-30 8:39 ` Christian König 2024-01-30 22:29 ` Zeng, Oak 2024-01-30 22:29 ` Zeng, Oak 2024-01-30 23:12 ` David Airlie 2024-01-30 23:12 ` David Airlie 2024-01-31 9:15 ` Daniel Vetter 2024-01-31 9:15 ` Daniel Vetter 2024-01-31 20:17 ` Zeng, Oak 2024-01-31 20:17 ` Zeng, Oak 2024-01-31 20:59 ` Zeng, Oak 2024-01-31 20:59 ` Zeng, Oak 2024-02-01 8:52 ` Christian König 2024-02-01 8:52 ` Christian König 2024-02-29 18:22 ` Zeng, Oak 2024-03-08 4:43 ` Zeng, Oak 2024-03-08 10:07 ` Christian König 2024-01-30 8:43 ` Thomas Hellström 2024-01-30 8:43 ` Thomas Hellström 2024-01-29 15:03 ` Felix Kuehling 2024-01-29 15:03 ` Felix Kuehling 2024-01-29 15:33 ` Christian König 2024-01-29 15:33 ` Christian König 2024-01-29 16:24 ` Felix Kuehling 2024-01-29 16:24 ` Felix Kuehling 2024-01-29 16:28 ` Christian König 2024-01-29 16:28 ` Christian König 2024-01-29 17:52 ` Felix Kuehling 2024-01-29 17:52 ` Felix Kuehling 2024-01-29 19:03 ` Christian König 2024-01-29 19:03 ` Christian König 2024-01-29 20:24 ` Felix Kuehling 2024-01-29 20:24 ` Felix Kuehling 2024-02-23 20:12 ` Zeng, Oak 2024-02-27 6:54 ` Christian König 2024-02-27 15:58 ` Zeng, Oak 2024-02-28 19:51 ` Zeng, Oak 2024-02-29 9:41 ` Christian König 2024-02-29 16:05 ` Zeng, Oak 2024-02-29 17:12 ` Thomas Hellström 2024-03-01 7:01 ` Christian König 2024-01-17 22:12 ` [PATCH 22/23] drm/xe/svm: Add DRM_XE_SVM kernel config entry Oak Zeng 2024-01-17 22:12 ` Oak Zeng 2024-01-17 22:12 ` [PATCH 23/23] drm/xe/svm: Add svm memory hints interface Oak Zeng 2024-01-17 22:12 ` Oak Zeng 2024-01-18 2:45 ` ✓ CI.Patch_applied: success for XeKmd basic SVM support Patchwork 2024-01-18 2:46 ` ✗ CI.checkpatch: warning " Patchwork 2024-01-18 2:46 ` ✗ CI.KUnit: failure " Patchwork
Reply instructions: You may reply publicly to this message via plain-text email using any one of the following methods: * Save the following mbox file, import it into your mail client, and reply-to-all from there: mbox Avoid top-posting and favor interleaved quoting: https://en.wikipedia.org/wiki/Posting_style#Interleaved_style * Reply using the --to, --cc, and --in-reply-to switches of git-send-email(1): git send-email \ --in-reply-to=20240117221223.18540-17-oak.zeng@intel.com \ --to=oak.zeng@intel.com \ --cc=Thomas.Hellstrom@linux.intel.com \ --cc=brian.welty@intel.com \ --cc=dri-devel@lists.freedesktop.org \ --cc=himal.prasad.ghimiray@intel.com \ --cc=intel-xe@lists.freedesktop.org \ --cc=krishnaiah.bommu@intel.com \ --cc=matthew.brost@intel.com \ --cc=niranjana.vishwanathapura@intel.com \ /path/to/YOUR_REPLY https://kernel.org/pub/software/scm/git/docs/git-send-email.html * If your mail client supports setting the In-Reply-To header via mailto: links, try the mailto: linkBe sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.