From: Matthew Wilcox <willy@infradead.org>
To: Alex Sierra <alex.sierra@amd.com>
Cc: rcampbell@nvidia.com, Felix.Kuehling@amd.com,
dri-devel@lists.freedesktop.org, linux-mm@kvack.org,
jglisse@redhat.com, amd-gfx@lists.freedesktop.org,
jgg@nvidia.com, akpm@linux-foundation.org, hch@lst.de
Subject: Re: [RFC PATCH v2 0/8] Support DEVICE_GENERIC memory in migrate_vma_*
Date: Tue, 8 Jun 2021 15:26:45 +0100 [thread overview]
Message-ID: <YL9+JW1MPsANqqq1@casper.infradead.org> (raw)
In-Reply-To: <20210607204226.7743-1-alex.sierra@amd.com>
On Mon, Jun 07, 2021 at 03:42:18PM -0500, Alex Sierra wrote:
> v1:
> https://lore.kernel.org/linux-mm/20210529064022.GB15834@lst.de/T/
Please copy and paste the rationale into followup patch series instead
of sending a link:
AMD is building a system architecture for the Frontier supercomputer with
a coherent interconnect between CPUs and GPUs. This hardware architecture
allows the CPUs to coherently access GPU device memory. We have hardware
in our labs and we are working with our partner HPE on the BIOS, firmware
and software for delivery to the DOE.
The system BIOS advertises the GPU device memory (aka VRAM) as SPM
(special purpose memory) in the UEFI system address map. The amdgpu driver
looks it up with lookup_resource and registers it with devmap as
MEMORY_DEVICE_GENERIC using devm_memremap_pages.
Now we're trying to migrate data to and from that memory using the
migrate_vma_* helpers so we can support page-based migration in our
unified memory allocations, while also supporting CPU access to those
pages.
This patch series makes a few changes to make MEMORY_DEVICE_GENERIC pages
behave correctly in the migrate_vma_* helpers. We are looking for feedback
about this approach. If we're close, what's needed to make our patches
acceptable upstream? If we're not close, any suggestions how else to
achieve what we are trying to do (i.e. page migration and coherent CPU
access to VRAM)?
This work is based on HMM and our SVM memory manager that was recently
upstreamed to Dave Airlie's drm-next branch
[https://cgit.freedesktop.org/drm/drm/log/?h=drm-next]. On top of that we
did some rework of our VRAM management for migrations to remove some
incorrect assumptions, allow partially successful migrations and GPU
memory mappings that mix pages in VRAM and system memory.
[https://patchwork.kernel.org/project/dri-devel/list/?series=489811]
> v2:
> This patch series version has merged "[RFC PATCH v3 0/2]
> mm: remove extra ZONE_DEVICE struct page refcount" patch series made by
> Ralph Campbell. It also applies at the top of these series, our changes
> to support device generic type in migration_vma helpers.
> This has been tested in systems with device memory that has coherent
> access by CPU.
>
> Also addresses the following feedback made in v1:
> - Isolate in one patch kernel/resource.c modification, based
> on Christoph's feedback.
> - Add helpers check for generic and private type to avoid
> duplicated long lines.
>
> I like to provide an overview of what each of the patches does in a series:
>
> Patches 1-2: Rebased Ralph Campbell's ZONE_DEVICE page refcounting patches
> Patch 3: Export lookup_resource
> Patches 4-5: AMDGPU driver changes to register and use DEVICE_GENERIC memory
> Patches 6-8: Handle DEVICE_GENERIC memory in migration helpers
>
> Alex Sierra (6):
> kernel: resource: lookup_resource as exported symbol
> drm/amdkfd: add SPM support for SVM
> drm/amdkfd: generic type as sys mem on migration to ram
> include/linux/mm.h: helpers to check zone device generic type
> mm: add generic type support to migrate_vma helpers
> mm: call pgmap->ops->page_free for DEVICE_GENERIC pages
>
> Ralph Campbell (2):
> ext4/xfs: add page refcount helper
> mm: remove extra ZONE_DEVICE struct page refcount
>
> arch/powerpc/kvm/book3s_hv_uvmem.c | 2 +-
> drivers/gpu/drm/amd/amdkfd/kfd_migrate.c | 15 ++++--
> drivers/gpu/drm/nouveau/nouveau_dmem.c | 2 +-
> fs/dax.c | 8 +--
> fs/ext4/inode.c | 5 +-
> fs/xfs/xfs_file.c | 4 +-
> include/linux/dax.h | 10 ++++
> include/linux/memremap.h | 7 +--
> include/linux/mm.h | 52 +++---------------
> kernel/resource.c | 2 +-
> lib/test_hmm.c | 2 +-
> mm/internal.h | 8 +++
> mm/memremap.c | 69 +++++++-----------------
> mm/migrate.c | 13 ++---
> mm/page_alloc.c | 3 ++
> mm/swap.c | 45 ++--------------
> 16 files changed, 83 insertions(+), 164 deletions(-)
>
> --
> 2.17.1
>
>
prev parent reply other threads:[~2021-06-08 14:27 UTC|newest]
Thread overview: 14+ messages / expand[flat|nested] mbox.gz Atom feed top
2021-06-07 20:42 [RFC PATCH v2 0/8] Support DEVICE_GENERIC memory in migrate_vma_* Alex Sierra
2021-06-07 20:42 ` [RFC PATCH v2 1/8] ext4/xfs: add page refcount helper Alex Sierra
2021-06-08 0:29 ` Liam Howlett
2021-06-08 14:33 ` Matthew Wilcox
2021-06-09 19:23 ` Matthew Wilcox
2021-06-14 15:26 ` Felix Kuehling
2021-06-07 20:42 ` [RFC PATCH v2 2/8] mm: remove extra ZONE_DEVICE struct page refcount Alex Sierra
2021-06-07 20:42 ` [RFC PATCH v2 3/8] kernel: resource: lookup_resource as exported symbol Alex Sierra
2021-06-07 20:42 ` [RFC PATCH v2 4/8] drm/amdkfd: add SPM support for SVM Alex Sierra
2021-06-07 20:42 ` [RFC PATCH v2 5/8] drm/amdkfd: generic type as sys mem on migration to ram Alex Sierra
2021-06-07 20:42 ` [RFC PATCH v2 6/8] include/linux/mm.h: helpers to check zone device generic type Alex Sierra
2021-06-07 20:42 ` [RFC PATCH v2 7/8] mm: add generic type support to migrate_vma helpers Alex Sierra
2021-06-07 20:42 ` [RFC PATCH v2 8/8] mm: call pgmap->ops->page_free for DEVICE_GENERIC pages Alex Sierra
2021-06-08 14:26 ` Matthew Wilcox [this message]
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=YL9+JW1MPsANqqq1@casper.infradead.org \
--to=willy@infradead.org \
--cc=Felix.Kuehling@amd.com \
--cc=akpm@linux-foundation.org \
--cc=alex.sierra@amd.com \
--cc=amd-gfx@lists.freedesktop.org \
--cc=dri-devel@lists.freedesktop.org \
--cc=hch@lst.de \
--cc=jgg@nvidia.com \
--cc=jglisse@redhat.com \
--cc=linux-mm@kvack.org \
--cc=rcampbell@nvidia.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).