From: Jonathan Cameron <Jonathan.Cameron@Huawei.com>
To: Yuquan Wang <wangyuquan1236@phytium.com.cn>
Cc: <20240508130252.00006367@huawei.com>, <qemu-devel@nongnu.org>,
<linux-cxl@vger.kernel.org>
Subject: Re: CXL numa error on arm64 qemu virt machine
Date: Fri, 10 May 2024 18:16:46 +0100 [thread overview]
Message-ID: <20240510181646.000019a1@Huawei.com> (raw)
In-Reply-To: <ZjyK1tI0TtbAKhmh@phytium.com.cn>
On Thu, 9 May 2024 16:35:34 +0800
Yuquan Wang <wangyuquan1236@phytium.com.cn> wrote:
> On Wed, May 08, 2024 at 01:02:52PM +0100, Jonathan Cameron wrote:
> >
> > > [ 0.000000] ACPI: SRAT: Node 0 PXM 0 [mem 0x40000000-0xbfffffff]
> > > [ 0.000000] ACPI: SRAT: Node 1 PXM 1 [mem 0xc0000000-0x13fffffff]
> > > [ 0.000000] ACPI: Unknown target node for memory at 0x10000000000, assuming node 0
> > > [ 0.000000] NUMA: Warning: invalid memblk node 16 [mem 0x0000000004000000-0x0000000007ffffff]
> > > [ 0.000000] NUMA: Faking a node at [mem 0x0000000004000000-0x000000013fffffff]
> > > [ 0.000000] NUMA: NODE_DATA [mem 0x13f7f89c0-0x13f7fafff]
> > >
> > > Previous discussion: https://lore.kernel.org/linux-cxl/20231011150620.0000212a@Huawei.com/
> > >
> > > root@debian-bullseye-arm64:~# cxl create-region -d decoder0.0 -t ram
> > > [ 68.653873] cxl region0: Bypassing cpu_cache_invalidate_memregion() for testing!
> > > [ 68.660568] Unknown target node for memory at 0x10000000000, assuming node 0
> >
> > You need a load of kernel changes for NUMA nodes to work correctly with
> > CXL memory on arm64 platforms. I have some working code but need to tidy
> > up a few corners that came up in an internal review earlier this week.
> >
> > I have some travel coming up so may take a week or so to get those out.
> >
> > Curiously that invalid memblk has nothing to do with the CXL fixed memory window
> > Could you check if that is happening for you without the CXL patches?
> >
>
> Thanks.
>
> I have checked it, the problem is caused by my bios firmware file. I
> change the bios file and the numa topology can works.
>
> BTW, if it is convenient, could you help to post the link of the patches of CXL
> memory NUMA nodes on arm64 platforms?
https://git.kernel.org/pub/scm/linux/kernel/git/jic23/cxl-staging.git/log/?h=arm-numa-fixes
I've run out of time to sort out cover letters and things + just before the merge
window is never a good time get anyone to pay attention to potentially controversial
patches. So for now I've thrown up a branch on kernel.org with Robert's
series of fixes of related code (that's queued in the ACPI tree for the merge window)
and Dan Williams (from several years ago) + my additions that 'work' (lightly tested)
on qemu/arm64 with the generic port patches etc.
I'll send out an RFC in a couple of weeks. In meantime let me know if you
run into any problems or have suggestions to improve them.
Jonathan
p.s. Apparently my computer is in the future. (-28 minutes and counting!)
>
> >
> > Whilst it doesn't work yet (because of missing kernel support)
> > you'll need something that looks more like the generic ports test added in
> > https://gitlab.com/jic23/qemu/-/commit/6589c527920ba22fe0923b60b58d33a8e9fd371e
> >
> > Most importantly
> > -numa node,nodeid=2 -object acpi-generic-port,id=gp0,pci-bus-cxl.1,node=2
> > + the bits setting distances etc. Note CXL memory does not provide SLIT like
> > data at the moment, so the test above won't help you identify if it is correctly
> > set up. That's a gap in general in the kernel support. Whilst we'd love
> > it if everyone moved to hmat derived information we may need to provide
> > some fallback.
> >
> > Jonathan
> >
>
> Many thanks
> Yuquan
>
next prev parent reply other threads:[~2024-05-10 17:16 UTC|newest]
Thread overview: 7+ messages / expand[flat|nested] mbox.gz Atom feed top
2024-05-08 8:00 CXL numa error on arm64 qemu virt machine Yuquan Wang
2024-05-08 12:02 ` Jonathan Cameron
2024-05-09 8:35 ` Yuquan Wang
2024-05-10 17:16 ` Jonathan Cameron [this message]
2024-05-17 10:07 ` Yuquan Wang
2024-05-17 10:14 ` Jonathan Cameron
2024-05-17 18:03 ` Jonathan Cameron
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20240510181646.000019a1@Huawei.com \
--to=jonathan.cameron@huawei.com \
--cc=20240508130252.00006367@huawei.com \
--cc=linux-cxl@vger.kernel.org \
--cc=qemu-devel@nongnu.org \
--cc=wangyuquan1236@phytium.com.cn \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).