* [PATCH linux-next v3] x86/xen/time: prefer tsc as clocksource when it is invariant
@ 2022-12-16 16:21 Krister Johansen
2023-01-17 12:12 ` Juergen Gross
2023-02-13 8:18 ` Juergen Gross
0 siblings, 2 replies; 3+ messages in thread
From: Krister Johansen @ 2022-12-16 16:21 UTC (permalink / raw
To: Juergen Gross, Boris Ostrovsky, Jan Beulich
Cc: Thomas Gleixner, Ingo Molnar, Borislav Petkov, Dave Hansen, x86,
H. Peter Anvin, xen-devel, linux-kernel, Marcelo Tosatti,
Anthony Liguori, David Reaver, Brendan Gregg
Kvm elects to use tsc instead of kvm-clock when it can detect that the
TSC is invariant.
(As of commit 7539b174aef4 ("x86: kvmguest: use TSC clocksource if
invariant TSC is exposed")).
Notable cloud vendors[1] and performance engineers[2] recommend that Xen
users preferentially select tsc over xen-clocksource due the performance
penalty incurred by the latter. These articles are persuasive and
tailored to specific use cases. In order to understand the tradeoffs
around this choice more fully, this author had to reference the
documented[3] complexities around the Xen configuration, as well as the
kernel's clocksource selection algorithm. Many users may not attempt
this to correctly configure the right clock source in their guest.
The approach taken in the kvm-clock module spares users this confusion,
where possible.
Both the Intel SDM[4] and the Xen tsc documentation explain that marking
a tsc as invariant means that it should be considered stable by the OS
and is elibile to be used as a wall clock source.
In order to obtain better out-of-the-box performance, and reduce the
need for user tuning, follow kvm's approach and decrease the xen clock
rating so that tsc is preferable, if it is invariant, stable, and the
tsc will never be emulated.
[1] https://aws.amazon.com/premiumsupport/knowledge-center/manage-ec2-linux-clock-source/
[2] https://www.brendangregg.com/blog/2021-09-26/the-speed-of-time.html
[3] https://xenbits.xen.org/docs/unstable/man/xen-tscmode.7.html
[4] Intel 64 and IA-32 Architectures Sofware Developer's Manual Volume
3b: System Programming Guide, Part 2, Section 17.17.1, Invariant TSC
Signed-off-by: Krister Johansen <kjlx@templeofstupid.com>
Code-reviewed-by: David Reaver <me@davidreaver.com>
---
v3:
- Temporarily drop references to public API headers
- tsc_emulated check is redundant if tsc_mode is no emulate. (feedback from
Boris Ostrovsky)
- Use cpuid_count instead of cpuid when accessing a cpuid leaf with subleaves.
(feedback from Jan Beulich)
- Kernel has its own cpuid hypercall and can intercept checks about the tsc
mode. The check for PV mode is inapposite and therefore removed. (feedback
from Jan Beulich)
v2:
- Use cpuid information to determine if tsc is emulated. Do not use tsc as
clocksource if it is. (feedback from Boris Ostrovsky)
- Move tsc checks into their own helper function
- Add defines for tsc cpuid flags needed by new helper function.
---
arch/x86/xen/time.c | 38 +++++++++++++++++++++++++++++++++++++-
1 file changed, 37 insertions(+), 1 deletion(-)
diff --git a/arch/x86/xen/time.c b/arch/x86/xen/time.c
index 9ef0a5cca96e..95140609c8a8 100644
--- a/arch/x86/xen/time.c
+++ b/arch/x86/xen/time.c
@@ -474,15 +474,51 @@ static void xen_setup_vsyscall_time_info(void)
xen_clocksource.vdso_clock_mode = VDSO_CLOCKMODE_PVCLOCK;
}
+/*
+ * Check if it is possible to safely use the tsc as a clocksource. This is
+ * only true if the hypervisor notifies the guest that its tsc is invariant,
+ * the tsc is stable, and the tsc instruction will never be emulated.
+ */
+static int __init xen_tsc_safe_clocksource(void)
+{
+ u32 eax, ebx, ecx, edx;
+
+ if (!(boot_cpu_has(X86_FEATURE_CONSTANT_TSC)))
+ return 0;
+
+ if (!(boot_cpu_has(X86_FEATURE_NONSTOP_TSC)))
+ return 0;
+
+ if (check_tsc_unstable())
+ return 0;
+
+ /* Leaf 4, sub-leaf 0 (0x40000x03) */
+ cpuid_count(xen_cpuid_base() + 3, 0, &eax, &ebx, &ecx, &edx);
+
+ /* tsc_mode = no_emulate (2) */
+ if (ebx != 2)
+ return 0;
+
+ return 1;
+}
+
static void __init xen_time_init(void)
{
struct pvclock_vcpu_time_info *pvti;
int cpu = smp_processor_id();
struct timespec64 tp;
- /* As Dom0 is never moved, no penalty on using TSC there */
+ /*
+ * As Dom0 is never moved, no penalty on using TSC there.
+ *
+ * If it is possible for the guest to determine that the tsc is a safe
+ * clocksource, then set xen_clocksource rating below that of the tsc
+ * so that the system prefers tsc instead.
+ */
if (xen_initial_domain())
xen_clocksource.rating = 275;
+ else if (xen_tsc_safe_clocksource())
+ xen_clocksource.rating = 299;
clocksource_register_hz(&xen_clocksource, NSEC_PER_SEC);
--
2.25.1
^ permalink raw reply related [flat|nested] 3+ messages in thread
* Re: [PATCH linux-next v3] x86/xen/time: prefer tsc as clocksource when it is invariant
2022-12-16 16:21 [PATCH linux-next v3] x86/xen/time: prefer tsc as clocksource when it is invariant Krister Johansen
@ 2023-01-17 12:12 ` Juergen Gross
2023-02-13 8:18 ` Juergen Gross
1 sibling, 0 replies; 3+ messages in thread
From: Juergen Gross @ 2023-01-17 12:12 UTC (permalink / raw
To: Krister Johansen, Boris Ostrovsky, Jan Beulich
Cc: Thomas Gleixner, Ingo Molnar, Borislav Petkov, Dave Hansen, x86,
H. Peter Anvin, xen-devel, linux-kernel, Marcelo Tosatti,
Anthony Liguori, David Reaver, Brendan Gregg
[-- Attachment #1.1.1: Type: text/plain, Size: 1962 bytes --]
On 16.12.22 17:21, Krister Johansen wrote:
> Kvm elects to use tsc instead of kvm-clock when it can detect that the
> TSC is invariant.
>
> (As of commit 7539b174aef4 ("x86: kvmguest: use TSC clocksource if
> invariant TSC is exposed")).
>
> Notable cloud vendors[1] and performance engineers[2] recommend that Xen
> users preferentially select tsc over xen-clocksource due the performance
> penalty incurred by the latter. These articles are persuasive and
> tailored to specific use cases. In order to understand the tradeoffs
> around this choice more fully, this author had to reference the
> documented[3] complexities around the Xen configuration, as well as the
> kernel's clocksource selection algorithm. Many users may not attempt
> this to correctly configure the right clock source in their guest.
>
> The approach taken in the kvm-clock module spares users this confusion,
> where possible.
>
> Both the Intel SDM[4] and the Xen tsc documentation explain that marking
> a tsc as invariant means that it should be considered stable by the OS
> and is elibile to be used as a wall clock source.
>
> In order to obtain better out-of-the-box performance, and reduce the
> need for user tuning, follow kvm's approach and decrease the xen clock
> rating so that tsc is preferable, if it is invariant, stable, and the
> tsc will never be emulated.
>
> [1] https://aws.amazon.com/premiumsupport/knowledge-center/manage-ec2-linux-clock-source/
> [2] https://www.brendangregg.com/blog/2021-09-26/the-speed-of-time.html
> [3] https://xenbits.xen.org/docs/unstable/man/xen-tscmode.7.html
> [4] Intel 64 and IA-32 Architectures Sofware Developer's Manual Volume
> 3b: System Programming Guide, Part 2, Section 17.17.1, Invariant TSC
>
> Signed-off-by: Krister Johansen <kjlx@templeofstupid.com>
> Code-reviewed-by: David Reaver <me@davidreaver.com>
Reviewed-by: Juergen Gross <jgross@suse.com>
Juergen
[-- Attachment #1.1.2: OpenPGP public key --]
[-- Type: application/pgp-keys, Size: 3149 bytes --]
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 495 bytes --]
^ permalink raw reply [flat|nested] 3+ messages in thread
* Re: [PATCH linux-next v3] x86/xen/time: prefer tsc as clocksource when it is invariant
2022-12-16 16:21 [PATCH linux-next v3] x86/xen/time: prefer tsc as clocksource when it is invariant Krister Johansen
2023-01-17 12:12 ` Juergen Gross
@ 2023-02-13 8:18 ` Juergen Gross
1 sibling, 0 replies; 3+ messages in thread
From: Juergen Gross @ 2023-02-13 8:18 UTC (permalink / raw
To: Krister Johansen, Boris Ostrovsky, Jan Beulich
Cc: Thomas Gleixner, Ingo Molnar, Borislav Petkov, Dave Hansen, x86,
H. Peter Anvin, xen-devel, linux-kernel, Marcelo Tosatti,
Anthony Liguori, David Reaver, Brendan Gregg
[-- Attachment #1.1.1: Type: text/plain, Size: 1954 bytes --]
On 16.12.22 17:21, Krister Johansen wrote:
> Kvm elects to use tsc instead of kvm-clock when it can detect that the
> TSC is invariant.
>
> (As of commit 7539b174aef4 ("x86: kvmguest: use TSC clocksource if
> invariant TSC is exposed")).
>
> Notable cloud vendors[1] and performance engineers[2] recommend that Xen
> users preferentially select tsc over xen-clocksource due the performance
> penalty incurred by the latter. These articles are persuasive and
> tailored to specific use cases. In order to understand the tradeoffs
> around this choice more fully, this author had to reference the
> documented[3] complexities around the Xen configuration, as well as the
> kernel's clocksource selection algorithm. Many users may not attempt
> this to correctly configure the right clock source in their guest.
>
> The approach taken in the kvm-clock module spares users this confusion,
> where possible.
>
> Both the Intel SDM[4] and the Xen tsc documentation explain that marking
> a tsc as invariant means that it should be considered stable by the OS
> and is elibile to be used as a wall clock source.
>
> In order to obtain better out-of-the-box performance, and reduce the
> need for user tuning, follow kvm's approach and decrease the xen clock
> rating so that tsc is preferable, if it is invariant, stable, and the
> tsc will never be emulated.
>
> [1] https://aws.amazon.com/premiumsupport/knowledge-center/manage-ec2-linux-clock-source/
> [2] https://www.brendangregg.com/blog/2021-09-26/the-speed-of-time.html
> [3] https://xenbits.xen.org/docs/unstable/man/xen-tscmode.7.html
> [4] Intel 64 and IA-32 Architectures Sofware Developer's Manual Volume
> 3b: System Programming Guide, Part 2, Section 17.17.1, Invariant TSC
>
> Signed-off-by: Krister Johansen <kjlx@templeofstupid.com>
> Code-reviewed-by: David Reaver <me@davidreaver.com>
Pushed to: xen/tip.git for-linus-6.3
Juergen
[-- Attachment #1.1.2: OpenPGP public key --]
[-- Type: application/pgp-keys, Size: 3149 bytes --]
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 495 bytes --]
^ permalink raw reply [flat|nested] 3+ messages in thread
end of thread, other threads:[~2023-02-13 8:18 UTC | newest]
Thread overview: 3+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2022-12-16 16:21 [PATCH linux-next v3] x86/xen/time: prefer tsc as clocksource when it is invariant Krister Johansen
2023-01-17 12:12 ` Juergen Gross
2023-02-13 8:18 ` Juergen Gross
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).