* S0ix support in Xen
@ 2023-02-27 11:48 Simon Gaiser
2023-02-27 12:11 ` Jan Beulich
` (2 more replies)
0 siblings, 3 replies; 5+ messages in thread
From: Simon Gaiser @ 2023-02-27 11:48 UTC (permalink / raw
To: xen-devel
[-- Attachment #1.1: Type: text/plain, Size: 3126 bytes --]
Hi,
I have been looking into using S0ix with Xen. On systems with with 11th
gen (Tiger Lake) Intel mobile CPUs or newer this is often the only
supported suspend method, thus we want to support it in Qubes OS.
Below a summary of my current understanding of what's needed (and known
unknowns). I would appreciate some feedback (what's missing, preferred
solutions, etc.).
Note this topic is much above my previous experience with Xen and x86
power management internals, so sorry if I'm missing things that are
obvious to you.
PIT timer: During some previous private discussion it was mentioned that
the PIT timer that Xen initializes for IO-APIC testing prevents S0ix
residency and therefore that part needs to be reworked. But if I'm
reading the current code correctly Xen can already use the HPET timer
instead, either with an automatic fallback if PIT is unavailable or by
forcing it via hpet=legacy-replacement=1. Looking at the rest I think
the PIT isn't used if Xen finds another clocksource. Did I miss
something?
mwait idle driver: While mwait-idle.c share a lot of code with Linux's
intel_idle.c and the imported code seems to have been updated (for
example for Alder Lake) it only supports the CPUs with hardcoded
cstates. Linux's code also has a code path to read the cstate config
from ACPI if the driver doesn't has a hard coded config for the model.
This is needed for example for Tiger Lake. For my current testing I
added the values the Linux code reads from ACPI by hand. But that's of
course no proper solution. How should this be handled in Xen (IIUC some
ACPI things are handled by Xen and some by dom0)?
The debugging code in xen/arch/x86/acpi/cpu_idle.c (and maybe other
places) need to learn about deeper package C states, for example for
Tiger Lake.
In general having the power management debugging available that you have
under plain Linux through /sys/kernel/debug/pmc_core, etc. would be
nice. Some things are as easy as allowing some dom0 to read some MSRs.
But didn't looked into it further, some functions might also be required
more complex integration (like extending xenpm's functionality).
I'm not sure yet is what else in Xen needs to learn about S0ix. Running
domains need to be halted, that can be handled by the toolstack. What
other Xen internal things need to be aware of S0ix? Like avoiding
unnecessary timer wakeups or similar. That's currently my biggest
unknown and it would be great if someone with more insight and overview
could give some hints here.
On my test system (NUC11TNHi5, Tiger Lake) I haven't seen it reaching
cstates lower PC7 with Xen (so it can of course not reach S0ix
residency). With plain Linux I got it working on that system although it
was rather fiddly and probably something is fishy on the firmware side.
After seeing it very occasionally not working under plain Linux I have
installed a newer firmware version. With that Xen currently doesn't wake
up after triggering s2idle in dom0. I'm currently looking into that and
will follow up if I have more details on that part.
Thanks, Simon
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: S0ix support in Xen
2023-02-27 11:48 S0ix support in Xen Simon Gaiser
@ 2023-02-27 12:11 ` Jan Beulich
2023-03-13 14:59 ` Simon Gaiser
2023-03-29 14:38 ` Roger Pau Monné
2 siblings, 0 replies; 5+ messages in thread
From: Jan Beulich @ 2023-02-27 12:11 UTC (permalink / raw
To: Simon Gaiser; +Cc: xen-devel
On 27.02.2023 12:48, Simon Gaiser wrote:
> PIT timer: During some previous private discussion it was mentioned that
> the PIT timer that Xen initializes for IO-APIC testing prevents S0ix
> residency and therefore that part needs to be reworked. But if I'm
> reading the current code correctly Xen can already use the HPET timer
> instead, either with an automatic fallback if PIT is unavailable or by
> forcing it via hpet=legacy-replacement=1. Looking at the rest I think
> the PIT isn't used if Xen finds another clocksource. Did I miss
> something?
I think the concern was with calibrating the APIC clock against PIT,
but as you say that connection has been cut. So I think there's no
prereq work left there; Andrew may prove me wrong, though.
> mwait idle driver: While mwait-idle.c share a lot of code with Linux's
> intel_idle.c and the imported code seems to have been updated (for
> example for Alder Lake) it only supports the CPUs with hardcoded
> cstates. Linux's code also has a code path to read the cstate config
> from ACPI if the driver doesn't has a hard coded config for the model.
> This is needed for example for Tiger Lake. For my current testing I
> added the values the Linux code reads from ACPI by hand. But that's of
> course no proper solution. How should this be handled in Xen (IIUC some
> ACPI things are handled by Xen and some by dom0)?
Well, there's first of all some lack of understanding on my part: It
hasn't become clear to me why Linux now has two CPU idle drivers
consuming ACPI data (intel_idle and the purely ACPI-based one). If
two are really needed, then I guess mwait-idle would need extending
to also consume data passed up from Dom0, just like the ACPI-only
driver in Xen does (and which doesn't do anything until such data
was supplied).
> I'm not sure yet is what else in Xen needs to learn about S0ix. Running
> domains need to be halted, that can be handled by the toolstack. What
> other Xen internal things need to be aware of S0ix? Like avoiding
> unnecessary timer wakeups or similar. That's currently my biggest
> unknown and it would be great if someone with more insight and overview
> could give some hints here.
I didn't think the higher-level-software side of things was different
from S3. Instead I thought it's merely a matter of different
interaction with hardware that's needed (which, as per above, includes
avoiding certain things, i.e. in particular the PIT).
Jan
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: S0ix support in Xen
2023-02-27 11:48 S0ix support in Xen Simon Gaiser
2023-02-27 12:11 ` Jan Beulich
@ 2023-03-13 14:59 ` Simon Gaiser
2023-03-29 14:38 ` Roger Pau Monné
2 siblings, 0 replies; 5+ messages in thread
From: Simon Gaiser @ 2023-03-13 14:59 UTC (permalink / raw
To: xen-devel
[-- Attachment #1.1: Type: text/plain, Size: 751 bytes --]
Simon Gaiser:
[...]
> On my test system (NUC11TNHi5, Tiger Lake) I haven't seen it reaching
> cstates lower PC7 with Xen (so it can of course not reach S0ix
> residency). With plain Linux I got it working on that system although it
> was rather fiddly and probably something is fishy on the firmware side.
> After seeing it very occasionally not working under plain Linux I have
> installed a newer firmware version. With that Xen currently doesn't wake
> up after triggering s2idle in dom0. I'm currently looking into that and
> will follow up if I have more details on that part.
Solved the wakeup issue, see
https://lore.kernel.org/xen-devel/20230313134102.3157-1-simon@invisiblethingslab.com/
(doesn't change the cstate it reaches)
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: S0ix support in Xen
2023-02-27 11:48 S0ix support in Xen Simon Gaiser
2023-02-27 12:11 ` Jan Beulich
2023-03-13 14:59 ` Simon Gaiser
@ 2023-03-29 14:38 ` Roger Pau Monné
2023-04-17 11:58 ` Simon Gaiser
2 siblings, 1 reply; 5+ messages in thread
From: Roger Pau Monné @ 2023-03-29 14:38 UTC (permalink / raw
To: Simon Gaiser; +Cc: xen-devel
On Mon, Feb 27, 2023 at 12:48:03PM +0100, Simon Gaiser wrote:
> Hi,
>
> I have been looking into using S0ix with Xen. On systems with with 11th
> gen (Tiger Lake) Intel mobile CPUs or newer this is often the only
> supported suspend method, thus we want to support it in Qubes OS.
>
> Below a summary of my current understanding of what's needed (and known
> unknowns). I would appreciate some feedback (what's missing, preferred
> solutions, etc.).
>
> Note this topic is much above my previous experience with Xen and x86
> power management internals, so sorry if I'm missing things that are
> obvious to you.
>
> PIT timer: During some previous private discussion it was mentioned that
> the PIT timer that Xen initializes for IO-APIC testing prevents S0ix
> residency and therefore that part needs to be reworked. But if I'm
> reading the current code correctly Xen can already use the HPET timer
> instead, either with an automatic fallback if PIT is unavailable or by
> forcing it via hpet=legacy-replacement=1. Looking at the rest I think
> the PIT isn't used if Xen finds another clocksource. Did I miss
> something?
Do you have some reference to documentation related to the S0ix
states?
I would like to understand exactly what's required in terms of
hardware devices the OS can use and still be able to enter such
states.
Thanks, Roger.
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: S0ix support in Xen
2023-03-29 14:38 ` Roger Pau Monné
@ 2023-04-17 11:58 ` Simon Gaiser
0 siblings, 0 replies; 5+ messages in thread
From: Simon Gaiser @ 2023-04-17 11:58 UTC (permalink / raw
To: Roger Pau Monné; +Cc: xen-devel
[-- Attachment #1.1: Type: text/plain, Size: 2656 bytes --]
Roger Pau Monné:
> On Mon, Feb 27, 2023 at 12:48:03PM +0100, Simon Gaiser wrote:
>> Hi,
>>
>> I have been looking into using S0ix with Xen. On systems with with
>> 11th gen (Tiger Lake) Intel mobile CPUs or newer this is often the
>> only supported suspend method, thus we want to support it in Qubes
>> OS.
>>
>> Below a summary of my current understanding of what's needed (and
>> known unknowns). I would appreciate some feedback (what's missing,
>> preferred solutions, etc.).
>>
>> Note this topic is much above my previous experience with Xen and x86
>> power management internals, so sorry if I'm missing things that are
>> obvious to you.
>>
>> PIT timer: During some previous private discussion it was mentioned
>> that the PIT timer that Xen initializes for IO-APIC testing prevents
>> S0ix residency and therefore that part needs to be reworked. But if
>> I'm reading the current code correctly Xen can already use the HPET
>> timer instead, either with an automatic fallback if PIT is
>> unavailable or by forcing it via hpet=legacy-replacement=1. Looking
>> at the rest I think the PIT isn't used if Xen finds another
>> clocksource. Did I miss something?
>
> Do you have some reference to documentation related to the S0ix
> states?
>
> I would like to understand exactly what's required in terms of
> hardware devices the OS can use and still be able to enter such
> states.
Unfortunately the documentation that I'm aware of is rather sparse.
There are two blog posts by Intel [1] and [2] that are quite good when
you are trying to get it working under Linux. [3] might be also useful
for debugging.
But I'm not aware of an explicit and complete list of what is required
to enter S0ix. You can deduct things from the blog posts or for example
the names of the debug register fields, but yeah ...
Also [2] contains this gem:
Currently, only the NDA version of the Intel® SoC Watch tool can
expose the IP Link Power State. [...] If you have any questions,
please refer to your Intel representative.
Regarding what devices are active in S0ix the PCH datasheet [4] has a
few additional pointers like:
USB 3.2 only works in S0. USB 2.0 survives S0ix and Sx states and
provides early boot access.
If someone knows about more public docs, I'm all ear.
Simon
[1]: https://01.org/blogs/qwang59/2018/how-achieve-s0ix-states-linux
[2]: https://01.org/blogs/qwang59/2020/linux-s0ix-troubleshooting
[3]: https://01.org/blogs/2019/using-power-management-controller-drivers-debug-low-power-platform-states
[4]: https://cdrdv2.intel.com/v1/dl/getContent/631119
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]
^ permalink raw reply [flat|nested] 5+ messages in thread
end of thread, other threads:[~2023-04-17 11:59 UTC | newest]
Thread overview: 5+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2023-02-27 11:48 S0ix support in Xen Simon Gaiser
2023-02-27 12:11 ` Jan Beulich
2023-03-13 14:59 ` Simon Gaiser
2023-03-29 14:38 ` Roger Pau Monné
2023-04-17 11:58 ` Simon Gaiser
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).