From: Hongjian Fan <hongjian.fan@seagate.com>
To: "linux-cxl@vger.kernel.org" <linux-cxl@vger.kernel.org>
Subject: Question on deferring dax registration to cxl module for CXL_REGION
Date: Thu, 11 Jan 2024 21:03:48 +0000 [thread overview]
Message-ID: <CH0PR20MB4250F264AA40C6E9CF1F599C90682@CH0PR20MB4250.namprd20.prod.outlook.com> (raw)
Hi CXL experts,
I have observed the following behavior on iomem_resourece when CONFIG_CXL_REGION is enabled in the kernel.
CXL windows are inserted into iomem_resourece based on CEDT CFMWS. If there is only one CXL device attached to the host, the CXL window matches the soft reserved memory range, and the CXL window is inserted as the child of the PCI mem and the parent of the soft reserve. But if there are multiple CXL windows, each of the CXL window is part of the soft reserved memory range, the CXL window is inserted as the child of the soft reserved memory.
Function dax_hmem_platform_probe defers the dax region registration for the CXL window to cxl module.
However, two issues seem to occur:
1) If the CXL window is not the direct child of the iomem_resourece, dax_hmem_platform_probe will not be able to detect and defer it. This means that if CFMWS contains multiple CXL windows, no deferral would happen.
2) If a CXL1.1 device is behind the CXL window, and the dax region registration is deferred. The dax region will not be created because CXL1.1 device doesn't have the HDM decoder and other features needed by the CXL module to create the dax region.
DAX ( and hmem ) module is not visible to the CXL device's features behind a CXL window, so it is impossible to defer only the CXL window for CXL2.0 devices.
If I want to make dax region show up when a single CXL1.1 device is attached, I can see two potential approaches:
1) Do not defer the CXL window in dax_hmem_platform_probe.
Can we simply not defer? Current code will not defer if multiple CXL windows presents. Is any issue observed when multiple CXL devices are attached?
2) Defer all CXL windows, and let cxl module create the dax region for CXL1.1 device.
But where should this creation be? It would be a long path to handle all the unvailable features from function cxl_pci_probe to reach function devm_cxl_add_dax_region.
Please provide your comments.
Below is the /proc/iomem output from my hardware:
1) When there is a single CXL2.0 device on the host, the CXL window is inserted in PCI mem and the soft reserved region is a child of the CXL window:
6080000000-707fffffff : CXL Window 0
6080000000-707fffffff : region0
6080000000-707fffffff : Soft Reserved
6080000000-707fffffff : dax0.0
6080000000-707fffffff : System RAM (kmem)
A cxl region is inserted under the CXL window by function discover_region and the dax region is registered by cxl_dax_region_probe
2) When there is a single CXL1.1 device on the host, it is similar but neither cxl region nor dax is created:
6080000000-707fffffff : CXL Window 0
6080000000-707fffffff : Soft Reserved
HDM decoder and other CXL2.0 features are missing from the CXL1.1 device so the CXL driver will not create related CXL structures. Because of the absence of the dax region, there is no numa node created for the cxl memory and the cxl memory is not usable in user space.
3) When there are multiple CXL devices, regardless CXL1.1 or 2.0, the CXL window is created under the soft reserved region:
6080000000-807fffffff : Soft Reserved
6080000000-707fffffff : CXL Window 0
6080000000-707fffffff : region0
6080000000-707fffffff : dax2.0
6080000000-707fffffff : System RAM (kmem)
7080000000-807fffffff : CXL Window 1
7080000000-807fffffff : dax3.0
7080000000-807fffffff : System RAM (kmem)
Both dax regions are registered by dax_hmem_platform_probe. The cxl region is created under CXL Window for the CXL2.0 devices.
Thanks,
Hongjian Fan
Seagate Internal
next reply other threads:[~2024-01-11 21:05 UTC|newest]
Thread overview: 5+ messages / expand[flat|nested] mbox.gz Atom feed top
2024-01-11 21:03 Hongjian Fan [this message]
2024-01-12 0:26 ` Question on deferring dax registration to cxl module for CXL_REGION Alison Schofield
2024-01-12 22:38 ` Hongjian Fan
2024-02-15 21:33 ` Nathan Fontenot
2024-02-29 21:52 ` Dan Williams
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=CH0PR20MB4250F264AA40C6E9CF1F599C90682@CH0PR20MB4250.namprd20.prod.outlook.com \
--to=hongjian.fan@seagate.com \
--cc=linux-cxl@vger.kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).