From: Yuquan Wang <wangyuquan1236@phytium.com.cn>
To: dan.j.williams@intel.com, jonathan.cameron@huawei.com,
y-goto@fujitsu.com
Cc: linux-cxl@vger.kernel.org, linux-kernel@vger.kernel.org,
qemu-devel@nongnu.org
Subject: RE: Questions about CXL device (type 3 memory) hotplug
Date: Wed, 27 Mar 2024 15:18:12 +0800 [thread overview]
Message-ID: <20240327071812.955794-1-wangyuquan1236@phytium.com.cn> (raw)
In-Reply-To: <646c04bbbd96_33fb32944b@dwillia2-xfh.jf.intel.com.notmuch>
On Mon, May 22, 2023 at 05:11:39PM -0700, Dan Williams wrote:
> Yasunori Gotou (Fujitsu) wrote:
[...]
Hi, all
There was some confusions about CXL device hotplug when I recently
tried to use Qemu to emulate CXL device hotplug and verify the relevant
functions of kernel.
> > Q1) Can PCIe hotplug driver detect and call CXL driver?
[...]
>
> Yes.
>
> The cxl_pci driver (drivers/cxl/pci.c) is just a typical PCI driver as
> far as the PCI hotplug driver is concerned. So add/remove events of a
> CXL card get turned into probe()/remove() events on the driver.
>
1. Can we divide steps of CXL device hotplug into two parts(PCI hotplug & Memory Hotplug)?
PCI Hotplug: the same as the native PCIe hotplug, including initializing cxl.io,
assigning PCIe BARs, allocating interrupts, etc. And the cxl_pci driver
is responsible for this part.
Memory Hotplug: focusing on enabling CXL memory including discovering and Configuring HDM,
extracting NUMA info from device, notifying memory management, etc.
> >
> > Q2) Can QEMU/KVM emulate CXL device hotplug?
> >
> > I heard that QEMU/KVM has PCIe device hotplug emulation, but I'm not sure
> > it can hotplug CXL device.
>
> It can, but as far as the driver is concerned you can achieve the same
> by:
>
> echo $devname > /sys/bus/pci/drivers/cxl_pci/unbind
>
> ...that exercises the same software flows as physical unplug.
>
2. What is the difference between "echo $devname > /sys/bus/pci/drivers/cxl_pci/unbind" and
"(qemu) device_del cxl-mem0" ?
According to the test, I found that "(qemu) device_del cxl-mem0" would directly
unplug the device and cause the interrupts on the cxl root port. It seems like this
operation would not only trigger cxl_pci driver but also pcieport driver.
The kernel dmesg is like below:
(qemu) device_del cxl-mem0
# dmesg
[ 699.057907] pcieport 0000:0c:00.0: pciehp: pending interrupts 0x0001 from Slot Status
[ 699.058929] pcieport 0000:0c:00.0: pciehp: Slot(0): Button press: will power off in 5 sec
[ 699.059986] pcieport 0000:0c:00.0: pciehp: pending interrupts 0x0010 from Slot Status
[ 699.060099] pcieport 0000:0c:00.0: pciehp: pciehp_set_indicators: SLOTCTRL 90 write cmd 2c0
Then I also tried "echo $devname > /sys/bus/pci/drivers/cxl_pci/unbind"
to check the behaviour of kernel. The kernel dmesg is like below:
# echo 0000:0d:00.0 > /sys/bus/pci/drivers/cxl_pci/unbind
# dmesg
[70387.978931] cxl_pci 0000:0d:00.0: vgaarb: pci_notify
[70388.021476] cxl_mem mem0: disconnect mem0 from port1
[70388.033099] pci 0000:0d:00.0: vgaarb: pci_notify
It seems like this operation would just unbind the cxl_pci driver from the cxl device.
Is my understanding about these two method correct?
3) Can I just use "ndctl/test/cxl-topology.sh" to test the cxl hotplug functions of kernel?
IIUC, cxl-topology.sh would utilize cxl_test (tools/testing/cxl) which is for regression
testing the kernel-user ABI.
PS: My qemu command line:
qemu-system-x86_64 \
-M q35,nvdimm=on,cxl=on \
-m 4G \
-smp 4 \
-object memory-backend-ram,size=2G,id=mem0 \
-numa node,nodeid=0,cpus=0-1,memdev=mem0 \
-object memory-backend-ram,size=2G,id=mem1 \
-numa node,nodeid=1,cpus=2-3,memdev=mem1 \
-object memory-backend-ram,size=256M,id=cxl-mem0 \
-device pxb-cxl,bus_nr=12,bus=pcie.0,id=cxl.1 \
-device cxl-rp,port=0,bus=cxl.1,id=root_port0,chassis=0,slot=0 \
-device cxl-type3,bus=root_port0,volatile-memdev=cxl-mem0,id=cxl-mem0 \
-M cxl-fmw.0.targets.0=cxl.1,cxl-fmw.0.size=4G,cxl-fmw.0.interleave-granularity=4k \
-hda ../disk/ubuntu_x86_test_new.qcow2 \
-nographic \
Qemu version: 8.2.50, the lastest commit of branch cxl-2024-03-05 in "https://gitlab.com/jic23/qemu"
Kernel version: 6.8.0-rc6
Many thanks
Yuquan
prev parent reply other threads:[~2024-03-27 7:18 UTC|newest]
Thread overview: 29+ messages / expand[flat|nested] mbox.gz Atom feed top
2023-05-22 8:06 Questions about CXL device (type 3 memory) hotplug Yasunori Gotou (Fujitsu)
2023-05-23 0:11 ` Dan Williams
2023-05-23 8:31 ` Yasunori Gotou (Fujitsu)
2023-05-23 17:36 ` Dan Williams
2023-05-24 11:12 ` Yasunori Gotou (Fujitsu)
2023-05-24 20:51 ` Dan Williams
2023-05-25 10:32 ` Yasunori Gotou (Fujitsu)
2023-05-26 8:05 ` Yasunori Gotou (Fujitsu)
2023-05-26 14:48 ` Dan Williams
2023-05-29 8:07 ` Yasunori Gotou (Fujitsu)
2023-06-06 17:58 ` Dan Williams
2023-06-08 7:39 ` Yasunori Gotou (Fujitsu)
2023-06-08 18:37 ` Dan Williams
2023-06-09 1:02 ` Yasunori Gotou (Fujitsu)
2023-05-23 13:34 ` Vikram Sethi
2023-05-23 18:40 ` Dan Williams
2023-05-24 0:02 ` Vikram Sethi
2023-05-24 4:03 ` Dan Williams
2023-05-24 14:47 ` Vikram Sethi
2023-05-24 21:20 ` Dan Williams
2023-05-31 4:25 ` Vikram Sethi
2023-06-06 20:54 ` Dan Williams
2023-06-07 1:06 ` Vikram Sethi
2023-06-07 15:12 ` Jonathan Cameron
2023-06-07 18:44 ` Vikram Sethi
2023-06-08 15:19 ` Jonathan Cameron
2023-06-08 18:41 ` Dan Williams
2024-03-27 7:10 ` Yuquan Wang
2024-03-27 7:18 ` Yuquan Wang [this message]
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20240327071812.955794-1-wangyuquan1236@phytium.com.cn \
--to=wangyuquan1236@phytium.com.cn \
--cc=dan.j.williams@intel.com \
--cc=jonathan.cameron@huawei.com \
--cc=linux-cxl@vger.kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=qemu-devel@nongnu.org \
--cc=y-goto@fujitsu.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).