From: Oak Zeng <oak.zeng@intel.com> To: dri-devel@lists.freedesktop.org, intel-xe@lists.freedesktop.org Cc: matthew.brost@intel.com, Thomas.Hellstrom@linux.intel.com, brian.welty@intel.com, himal.prasad.ghimiray@intel.com, krishnaiah.bommu@intel.com, niranjana.vishwanathapura@intel.com Subject: [PATCH 00/23] XeKmd basic SVM support Date: Wed, 17 Jan 2024 17:12:00 -0500 [thread overview] Message-ID: <20240117221223.18540-1-oak.zeng@intel.com> (raw) This is the very basic SVM (shared virtual memory) support in XeKmd driver. SVM allows the programmer to use a shaed virtual address space between CPU program and GPU program. It abstracts away from the user the location of the backing memory in a mixed CPU and GPU programming environment. This work is based on previous I915 SVM implementation mainly from Niranjana Vishwanathapura and Oak Zeng, which has never been upstreamed before. This is our first attempt to upstream this work. This implementation depends on Linux kernel HMM support. See some key designs in patch #1. We are aware there are currently some effort to implement SVM using GMEM(generalized memory management, see https://lore.kernel.org/dri-devel/20231128125025.4449-1-weixi.zhu@huawei.com/) We are open to this new method if it can be merged to upstream kernel. Before that, we think it is still safer to support SVM through HMM. This series only has basic SVM support. We think it is better to post this series earlier so we can get more eyes on it. Below are the works that is planned or ongoing: *Testing: We are working on the igt test right now. Some part of this series, especially the gpu page table update(patch #7, #8) and migration function (patch #10) need some debug to make it work. *Virtual address range based memory attributes and hints: We plan to expose uAPI for user to set memory attributes such as preferred location or migration granularity etc to a virtual address range. This is important to tune SVM performance. *GPU vram eviction: One key design choice of this series is, SVM layer allocate GPU memory directly from drm buddy allocator, instead of from xe vram manager. There is no BO (buffer object) concept in this implementation. The key benefit of this approach is we can migrate memory at page granularity easily. This also means SVM bypasses TTM's memory eviction logic. But we want the SVM memory and BO driver memory can mutually evicted each other. We have some prove of concept work to rework TTM resource manager for this purpose, see https://lore.kernel.org/dri-devel/20231102043306.2931989-1-oak.zeng@intel.com/ We will continue work on that series then implement SVM's eviction function based on the concept of shared drm LRU list b/t SVM and TTM/BO driver. Oak Zeng (23): drm/xe/svm: Add SVM document drm/xe/svm: Add svm key data structures drm/xe/svm: create xe svm during vm creation drm/xe/svm: Trace svm creation drm/xe/svm: add helper to retrieve svm range from address drm/xe/svm: Introduce a helper to build sg table from hmm range drm/xe/svm: Add helper for binding hmm range to gpu drm/xe/svm: Add helper to invalidate svm range from GPU drm/xe/svm: Remap and provide memmap backing for GPU vram drm/xe/svm: Introduce svm migration function drm/xe/svm: implement functions to allocate and free device memory drm/xe/svm: Trace buddy block allocation and free drm/xe/svm: Handle CPU page fault drm/xe/svm: trace svm range migration drm/xe/svm: Implement functions to register and unregister mmu notifier drm/xe/svm: Implement the mmu notifier range invalidate callback drm/xe/svm: clean up svm range during process exit drm/xe/svm: Move a few structures to xe_gt.h drm/xe/svm: migrate svm range to vram drm/xe/svm: Populate svm range drm/xe/svm: GPU page fault support drm/xe/svm: Add DRM_XE_SVM kernel config entry drm/xe/svm: Add svm memory hints interface Documentation/gpu/xe/index.rst | 1 + Documentation/gpu/xe/xe_svm.rst | 8 + drivers/gpu/drm/xe/Kconfig | 22 ++ drivers/gpu/drm/xe/Makefile | 5 + drivers/gpu/drm/xe/xe_device_types.h | 20 ++ drivers/gpu/drm/xe/xe_gt.h | 20 ++ drivers/gpu/drm/xe/xe_gt_pagefault.c | 28 +-- drivers/gpu/drm/xe/xe_migrate.c | 213 +++++++++++++++++ drivers/gpu/drm/xe/xe_migrate.h | 7 + drivers/gpu/drm/xe/xe_mmio.c | 12 + drivers/gpu/drm/xe/xe_pt.c | 147 +++++++++++- drivers/gpu/drm/xe/xe_pt.h | 5 + drivers/gpu/drm/xe/xe_svm.c | 324 +++++++++++++++++++++++++ drivers/gpu/drm/xe/xe_svm.h | 115 +++++++++ drivers/gpu/drm/xe/xe_svm_devmem.c | 232 ++++++++++++++++++ drivers/gpu/drm/xe/xe_svm_doc.h | 121 ++++++++++ drivers/gpu/drm/xe/xe_svm_migrate.c | 345 +++++++++++++++++++++++++++ drivers/gpu/drm/xe/xe_svm_range.c | 227 ++++++++++++++++++ drivers/gpu/drm/xe/xe_trace.h | 71 +++++- drivers/gpu/drm/xe/xe_vm.c | 7 + drivers/gpu/drm/xe/xe_vm_types.h | 15 +- include/uapi/drm/xe_drm.h | 40 ++++ 22 files changed, 1957 insertions(+), 28 deletions(-) create mode 100644 Documentation/gpu/xe/xe_svm.rst create mode 100644 drivers/gpu/drm/xe/xe_svm.c create mode 100644 drivers/gpu/drm/xe/xe_svm.h create mode 100644 drivers/gpu/drm/xe/xe_svm_devmem.c create mode 100644 drivers/gpu/drm/xe/xe_svm_doc.h create mode 100644 drivers/gpu/drm/xe/xe_svm_migrate.c create mode 100644 drivers/gpu/drm/xe/xe_svm_range.c -- 2.26.3
WARNING: multiple messages have this Message-ID (diff)
From: Oak Zeng <oak.zeng@intel.com> To: dri-devel@lists.freedesktop.org, intel-xe@lists.freedesktop.org Subject: [PATCH 00/23] XeKmd basic SVM support Date: Wed, 17 Jan 2024 17:12:00 -0500 [thread overview] Message-ID: <20240117221223.18540-1-oak.zeng@intel.com> (raw) This is the very basic SVM (shared virtual memory) support in XeKmd driver. SVM allows the programmer to use a shaed virtual address space between CPU program and GPU program. It abstracts away from the user the location of the backing memory in a mixed CPU and GPU programming environment. This work is based on previous I915 SVM implementation mainly from Niranjana Vishwanathapura and Oak Zeng, which has never been upstreamed before. This is our first attempt to upstream this work. This implementation depends on Linux kernel HMM support. See some key designs in patch #1. We are aware there are currently some effort to implement SVM using GMEM(generalized memory management, see https://lore.kernel.org/dri-devel/20231128125025.4449-1-weixi.zhu@huawei.com/) We are open to this new method if it can be merged to upstream kernel. Before that, we think it is still safer to support SVM through HMM. This series only has basic SVM support. We think it is better to post this series earlier so we can get more eyes on it. Below are the works that is planned or ongoing: *Testing: We are working on the igt test right now. Some part of this series, especially the gpu page table update(patch #7, #8) and migration function (patch #10) need some debug to make it work. *Virtual address range based memory attributes and hints: We plan to expose uAPI for user to set memory attributes such as preferred location or migration granularity etc to a virtual address range. This is important to tune SVM performance. *GPU vram eviction: One key design choice of this series is, SVM layer allocate GPU memory directly from drm buddy allocator, instead of from xe vram manager. There is no BO (buffer object) concept in this implementation. The key benefit of this approach is we can migrate memory at page granularity easily. This also means SVM bypasses TTM's memory eviction logic. But we want the SVM memory and BO driver memory can mutually evicted each other. We have some prove of concept work to rework TTM resource manager for this purpose, see https://lore.kernel.org/dri-devel/20231102043306.2931989-1-oak.zeng@intel.com/ We will continue work on that series then implement SVM's eviction function based on the concept of shared drm LRU list b/t SVM and TTM/BO driver. Oak Zeng (23): drm/xe/svm: Add SVM document drm/xe/svm: Add svm key data structures drm/xe/svm: create xe svm during vm creation drm/xe/svm: Trace svm creation drm/xe/svm: add helper to retrieve svm range from address drm/xe/svm: Introduce a helper to build sg table from hmm range drm/xe/svm: Add helper for binding hmm range to gpu drm/xe/svm: Add helper to invalidate svm range from GPU drm/xe/svm: Remap and provide memmap backing for GPU vram drm/xe/svm: Introduce svm migration function drm/xe/svm: implement functions to allocate and free device memory drm/xe/svm: Trace buddy block allocation and free drm/xe/svm: Handle CPU page fault drm/xe/svm: trace svm range migration drm/xe/svm: Implement functions to register and unregister mmu notifier drm/xe/svm: Implement the mmu notifier range invalidate callback drm/xe/svm: clean up svm range during process exit drm/xe/svm: Move a few structures to xe_gt.h drm/xe/svm: migrate svm range to vram drm/xe/svm: Populate svm range drm/xe/svm: GPU page fault support drm/xe/svm: Add DRM_XE_SVM kernel config entry drm/xe/svm: Add svm memory hints interface Documentation/gpu/xe/index.rst | 1 + Documentation/gpu/xe/xe_svm.rst | 8 + drivers/gpu/drm/xe/Kconfig | 22 ++ drivers/gpu/drm/xe/Makefile | 5 + drivers/gpu/drm/xe/xe_device_types.h | 20 ++ drivers/gpu/drm/xe/xe_gt.h | 20 ++ drivers/gpu/drm/xe/xe_gt_pagefault.c | 28 +-- drivers/gpu/drm/xe/xe_migrate.c | 213 +++++++++++++++++ drivers/gpu/drm/xe/xe_migrate.h | 7 + drivers/gpu/drm/xe/xe_mmio.c | 12 + drivers/gpu/drm/xe/xe_pt.c | 147 +++++++++++- drivers/gpu/drm/xe/xe_pt.h | 5 + drivers/gpu/drm/xe/xe_svm.c | 324 +++++++++++++++++++++++++ drivers/gpu/drm/xe/xe_svm.h | 115 +++++++++ drivers/gpu/drm/xe/xe_svm_devmem.c | 232 ++++++++++++++++++ drivers/gpu/drm/xe/xe_svm_doc.h | 121 ++++++++++ drivers/gpu/drm/xe/xe_svm_migrate.c | 345 +++++++++++++++++++++++++++ drivers/gpu/drm/xe/xe_svm_range.c | 227 ++++++++++++++++++ drivers/gpu/drm/xe/xe_trace.h | 71 +++++- drivers/gpu/drm/xe/xe_vm.c | 7 + drivers/gpu/drm/xe/xe_vm_types.h | 15 +- include/uapi/drm/xe_drm.h | 40 ++++ 22 files changed, 1957 insertions(+), 28 deletions(-) create mode 100644 Documentation/gpu/xe/xe_svm.rst create mode 100644 drivers/gpu/drm/xe/xe_svm.c create mode 100644 drivers/gpu/drm/xe/xe_svm.h create mode 100644 drivers/gpu/drm/xe/xe_svm_devmem.c create mode 100644 drivers/gpu/drm/xe/xe_svm_doc.h create mode 100644 drivers/gpu/drm/xe/xe_svm_migrate.c create mode 100644 drivers/gpu/drm/xe/xe_svm_range.c -- 2.26.3
next reply other threads:[~2024-01-17 22:01 UTC|newest] Thread overview: 198+ messages / expand[flat|nested] mbox.gz Atom feed top 2024-01-17 22:12 Oak Zeng [this message] 2024-01-17 22:12 ` [PATCH 00/23] XeKmd basic SVM support Oak Zeng 2024-01-17 22:12 ` [PATCH 01/23] drm/xe/svm: Add SVM document Oak Zeng 2024-01-17 22:12 ` Oak Zeng 2024-01-17 22:12 ` [PATCH 02/23] drm/xe/svm: Add svm key data structures Oak Zeng 2024-01-17 22:12 ` Oak Zeng 2024-01-17 22:12 ` [PATCH 03/23] drm/xe/svm: create xe svm during vm creation Oak Zeng 2024-01-17 22:12 ` Oak Zeng 2024-01-17 22:12 ` [PATCH 04/23] drm/xe/svm: Trace svm creation Oak Zeng 2024-01-17 22:12 ` Oak Zeng 2024-01-17 22:12 ` [PATCH 05/23] drm/xe/svm: add helper to retrieve svm range from address Oak Zeng 2024-01-17 22:12 ` Oak Zeng 2024-01-17 22:12 ` [PATCH 06/23] drm/xe/svm: Introduce a helper to build sg table from hmm range Oak Zeng 2024-01-17 22:12 ` Oak Zeng 2024-04-05 0:39 ` Jason Gunthorpe 2024-04-05 3:33 ` Zeng, Oak 2024-04-05 12:37 ` Jason Gunthorpe 2024-04-05 16:42 ` Zeng, Oak 2024-04-05 18:02 ` Jason Gunthorpe 2024-04-09 16:45 ` Zeng, Oak 2024-04-09 17:24 ` Jason Gunthorpe 2024-04-23 21:17 ` Zeng, Oak 2024-04-24 2:31 ` Matthew Brost 2024-04-24 13:57 ` Jason Gunthorpe 2024-04-24 16:35 ` Matthew Brost 2024-04-24 16:44 ` Jason Gunthorpe 2024-04-24 16:56 ` Matthew Brost 2024-04-24 17:48 ` Jason Gunthorpe 2024-04-24 13:48 ` Jason Gunthorpe 2024-04-24 23:59 ` Zeng, Oak 2024-04-25 1:05 ` Jason Gunthorpe 2024-04-26 9:55 ` Thomas Hellström 2024-04-26 12:00 ` Jason Gunthorpe 2024-04-26 14:49 ` Thomas Hellström 2024-04-26 16:35 ` Jason Gunthorpe 2024-04-29 8:25 ` Thomas Hellström 2024-04-30 17:30 ` Jason Gunthorpe 2024-04-30 18:57 ` Daniel Vetter 2024-05-01 0:09 ` Jason Gunthorpe 2024-05-02 8:04 ` Daniel Vetter 2024-05-02 9:11 ` Thomas Hellström 2024-05-02 12:46 ` Jason Gunthorpe 2024-05-02 15:01 ` Thomas Hellström 2024-05-02 19:25 ` Zeng, Oak 2024-05-03 13:37 ` Jason Gunthorpe 2024-05-03 14:43 ` Zeng, Oak 2024-05-03 16:28 ` Jason Gunthorpe 2024-05-03 20:29 ` Zeng, Oak 2024-05-04 1:03 ` Dave Airlie 2024-05-06 13:04 ` Daniel Vetter 2024-05-06 23:50 ` Matthew Brost 2024-05-07 11:56 ` Jason Gunthorpe 2024-05-06 13:33 ` Jason Gunthorpe 2024-04-09 17:33 ` Matthew Brost 2024-01-17 22:12 ` [PATCH 07/23] drm/xe/svm: Add helper for binding hmm range to gpu Oak Zeng 2024-01-17 22:12 ` Oak Zeng 2024-01-17 22:12 ` [PATCH 08/23] drm/xe/svm: Add helper to invalidate svm range from GPU Oak Zeng 2024-01-17 22:12 ` Oak Zeng 2024-01-17 22:12 ` [PATCH 09/23] drm/xe/svm: Remap and provide memmap backing for GPU vram Oak Zeng 2024-01-17 22:12 ` Oak Zeng 2024-01-17 22:12 ` [PATCH 10/23] drm/xe/svm: Introduce svm migration function Oak Zeng 2024-01-17 22:12 ` Oak Zeng 2024-01-17 22:12 ` [PATCH 11/23] drm/xe/svm: implement functions to allocate and free device memory Oak Zeng 2024-01-17 22:12 ` Oak Zeng 2024-01-17 22:12 ` [PATCH 12/23] drm/xe/svm: Trace buddy block allocation and free Oak Zeng 2024-01-17 22:12 ` Oak Zeng 2024-01-17 22:12 ` [PATCH 13/23] drm/xe/svm: Handle CPU page fault Oak Zeng 2024-01-17 22:12 ` Oak Zeng 2024-01-17 22:12 ` [PATCH 14/23] drm/xe/svm: trace svm range migration Oak Zeng 2024-01-17 22:12 ` Oak Zeng 2024-01-17 22:12 ` [PATCH 15/23] drm/xe/svm: Implement functions to register and unregister mmu notifier Oak Zeng 2024-01-17 22:12 ` Oak Zeng 2024-01-17 22:12 ` [PATCH 16/23] drm/xe/svm: Implement the mmu notifier range invalidate callback Oak Zeng 2024-01-17 22:12 ` Oak Zeng 2024-01-17 22:12 ` [PATCH 17/23] drm/xe/svm: clean up svm range during process exit Oak Zeng 2024-01-17 22:12 ` Oak Zeng 2024-01-17 22:12 ` [PATCH 18/23] drm/xe/svm: Move a few structures to xe_gt.h Oak Zeng 2024-01-17 22:12 ` Oak Zeng 2024-01-17 22:12 ` [PATCH 19/23] drm/xe/svm: migrate svm range to vram Oak Zeng 2024-01-17 22:12 ` Oak Zeng 2024-01-17 22:12 ` [PATCH 20/23] drm/xe/svm: Populate svm range Oak Zeng 2024-01-17 22:12 ` Oak Zeng 2024-01-17 22:12 ` [PATCH 21/23] drm/xe/svm: GPU page fault support Oak Zeng 2024-01-17 22:12 ` Oak Zeng 2024-01-23 2:06 ` Welty, Brian 2024-01-23 2:06 ` Welty, Brian 2024-01-23 3:09 ` Zeng, Oak 2024-01-23 3:09 ` Zeng, Oak 2024-01-23 3:21 ` Making drm_gpuvm work across gpu devices Zeng, Oak 2024-01-23 3:21 ` Zeng, Oak 2024-01-23 11:13 ` Christian König 2024-01-23 11:13 ` Christian König 2024-01-23 19:37 ` Zeng, Oak 2024-01-23 19:37 ` Zeng, Oak 2024-01-23 20:17 ` Felix Kuehling 2024-01-23 20:17 ` Felix Kuehling 2024-01-25 1:39 ` Zeng, Oak 2024-01-25 1:39 ` Zeng, Oak 2024-01-23 23:56 ` Danilo Krummrich 2024-01-23 23:56 ` Danilo Krummrich 2024-01-24 3:57 ` Zeng, Oak 2024-01-24 3:57 ` Zeng, Oak 2024-01-24 4:14 ` Zeng, Oak 2024-01-24 4:14 ` Zeng, Oak 2024-01-24 6:48 ` Christian König 2024-01-24 6:48 ` Christian König 2024-01-25 22:13 ` Danilo Krummrich 2024-01-25 22:13 ` Danilo Krummrich 2024-01-24 8:33 ` Christian König 2024-01-24 8:33 ` Christian König 2024-01-25 1:17 ` Zeng, Oak 2024-01-25 1:17 ` Zeng, Oak 2024-01-25 1:25 ` David Airlie 2024-01-25 1:25 ` David Airlie 2024-01-25 5:25 ` Zeng, Oak 2024-01-25 5:25 ` Zeng, Oak 2024-01-26 10:09 ` Christian König 2024-01-26 10:09 ` Christian König 2024-01-26 20:13 ` Zeng, Oak 2024-01-26 20:13 ` Zeng, Oak 2024-01-29 10:10 ` Christian König 2024-01-29 10:10 ` Christian König 2024-01-29 20:09 ` Zeng, Oak 2024-01-29 20:09 ` Zeng, Oak 2024-01-25 11:00 ` 回复:Making " 周春明(日月) 2024-01-25 11:00 ` 周春明(日月) 2024-01-25 17:00 ` Zeng, Oak 2024-01-25 17:00 ` Zeng, Oak 2024-01-25 17:15 ` Making " Felix Kuehling 2024-01-25 17:15 ` Felix Kuehling 2024-01-25 18:37 ` Zeng, Oak 2024-01-25 18:37 ` Zeng, Oak 2024-01-26 13:23 ` Christian König 2024-01-26 13:23 ` Christian König 2024-01-25 16:42 ` Zeng, Oak 2024-01-25 16:42 ` Zeng, Oak 2024-01-25 18:32 ` Daniel Vetter 2024-01-25 18:32 ` Daniel Vetter 2024-01-25 21:02 ` Zeng, Oak 2024-01-25 21:02 ` Zeng, Oak 2024-01-26 8:21 ` Thomas Hellström 2024-01-26 8:21 ` Thomas Hellström 2024-01-26 12:52 ` Christian König 2024-01-26 12:52 ` Christian König 2024-01-27 2:21 ` Zeng, Oak 2024-01-27 2:21 ` Zeng, Oak 2024-01-29 10:19 ` Christian König 2024-01-29 10:19 ` Christian König 2024-01-30 0:21 ` Zeng, Oak 2024-01-30 0:21 ` Zeng, Oak 2024-01-30 8:39 ` Christian König 2024-01-30 8:39 ` Christian König 2024-01-30 22:29 ` Zeng, Oak 2024-01-30 22:29 ` Zeng, Oak 2024-01-30 23:12 ` David Airlie 2024-01-30 23:12 ` David Airlie 2024-01-31 9:15 ` Daniel Vetter 2024-01-31 9:15 ` Daniel Vetter 2024-01-31 20:17 ` Zeng, Oak 2024-01-31 20:17 ` Zeng, Oak 2024-01-31 20:59 ` Zeng, Oak 2024-01-31 20:59 ` Zeng, Oak 2024-02-01 8:52 ` Christian König 2024-02-01 8:52 ` Christian König 2024-02-29 18:22 ` Zeng, Oak 2024-03-08 4:43 ` Zeng, Oak 2024-03-08 10:07 ` Christian König 2024-01-30 8:43 ` Thomas Hellström 2024-01-30 8:43 ` Thomas Hellström 2024-01-29 15:03 ` Felix Kuehling 2024-01-29 15:03 ` Felix Kuehling 2024-01-29 15:33 ` Christian König 2024-01-29 15:33 ` Christian König 2024-01-29 16:24 ` Felix Kuehling 2024-01-29 16:24 ` Felix Kuehling 2024-01-29 16:28 ` Christian König 2024-01-29 16:28 ` Christian König 2024-01-29 17:52 ` Felix Kuehling 2024-01-29 17:52 ` Felix Kuehling 2024-01-29 19:03 ` Christian König 2024-01-29 19:03 ` Christian König 2024-01-29 20:24 ` Felix Kuehling 2024-01-29 20:24 ` Felix Kuehling 2024-02-23 20:12 ` Zeng, Oak 2024-02-27 6:54 ` Christian König 2024-02-27 15:58 ` Zeng, Oak 2024-02-28 19:51 ` Zeng, Oak 2024-02-29 9:41 ` Christian König 2024-02-29 16:05 ` Zeng, Oak 2024-02-29 17:12 ` Thomas Hellström 2024-03-01 7:01 ` Christian König 2024-01-17 22:12 ` [PATCH 22/23] drm/xe/svm: Add DRM_XE_SVM kernel config entry Oak Zeng 2024-01-17 22:12 ` Oak Zeng 2024-01-17 22:12 ` [PATCH 23/23] drm/xe/svm: Add svm memory hints interface Oak Zeng 2024-01-17 22:12 ` Oak Zeng 2024-01-18 2:45 ` ✓ CI.Patch_applied: success for XeKmd basic SVM support Patchwork 2024-01-18 2:46 ` ✗ CI.checkpatch: warning " Patchwork 2024-01-18 2:46 ` ✗ CI.KUnit: failure " Patchwork
Reply instructions: You may reply publicly to this message via plain-text email using any one of the following methods: * Save the following mbox file, import it into your mail client, and reply-to-all from there: mbox Avoid top-posting and favor interleaved quoting: https://en.wikipedia.org/wiki/Posting_style#Interleaved_style * Reply using the --to, --cc, and --in-reply-to switches of git-send-email(1): git send-email \ --in-reply-to=20240117221223.18540-1-oak.zeng@intel.com \ --to=oak.zeng@intel.com \ --cc=Thomas.Hellstrom@linux.intel.com \ --cc=brian.welty@intel.com \ --cc=dri-devel@lists.freedesktop.org \ --cc=himal.prasad.ghimiray@intel.com \ --cc=intel-xe@lists.freedesktop.org \ --cc=krishnaiah.bommu@intel.com \ --cc=matthew.brost@intel.com \ --cc=niranjana.vishwanathapura@intel.com \ /path/to/YOUR_REPLY https://kernel.org/pub/software/scm/git/docs/git-send-email.html * If your mail client supports setting the In-Reply-To header via mailto: links, try the mailto: linkBe sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.