From: Ingo Molnar <mingo@kernel.org>
To: Vincent Guittot <vincent.guittot@linaro.org>
Cc: Linus Torvalds <torvalds@linux-foundation.org>,
linux-kernel@vger.kernel.org,
Peter Zijlstra <peterz@infradead.org>,
Thomas Gleixner <tglx@linutronix.de>,
Juri Lelli <juri.lelli@redhat.com>,
Dietmar Eggemann <dietmar.eggemann@arm.com>,
Steven Rostedt <rostedt@goodmis.org>,
Ben Segall <bsegall@google.com>, Mel Gorman <mgorman@suse.de>,
Daniel Bristot de Oliveira <bristot@redhat.com>,
Valentin Schneider <vschneid@redhat.com>
Subject: Re: [PATCH] Revert "sched/cpufreq: Rework schedutil governor performance estimation" and dependent commit
Date: Fri, 12 Jan 2024 19:24:14 +0100 [thread overview]
Message-ID: <ZaGDziEnKJ988zHh@gmail.com> (raw)
In-Reply-To: <ZaBUFETV53cRF7RB@gmail.com>
* Ingo Molnar <mingo@kernel.org> wrote:
> > I can provide a clean revert of only :
> > f12560779f9d ("sched/cpufreq: Rework iowait boost")
> > 9c0b4bb7f630 ("sched/cpufreq: Rework schedutil governor performance estimation")
>
> I've done this too, see this new commit in sched/urgent:
>
> 60ee1706bd11 ("Revert "sched/cpufreq: Rework schedutil governor performance estimation" and dependent commit")
>
> Also attached below.
>
> > if the fix that i proposed doesn't work:
> > https://lore.kernel.org/all/ZZ+ixagkxRPYyTCE@vingu-book/
>
> Yeah - although of course Linus is free to just pull the revert as well.
> I'll try to reproduce the regression locally as well.
Update & heads up: unfortunately I'm unable to reproduce the regression on
a fairly similar system with a Threadripper 3970X CPU.
Kernel build times are very close, with or without the revert, on vanilla
v6.7 or v6.7+sched/core.
Here's a few results where I tried to quantify kernel build times without
having to wait a long time.
Re-building the kernel/**.o object files in a loop:
$ perf stat --pre 'rm -f kernel/*.o kernel/*/*.o kernel/*/*/*.o' --null --sync --repeat 3 make -j64 kernel/ >/dev/null
# v6.7.0:
# bootup default schedutil governor:
24.521 +- 0.077 seconds time elapsed ( +- 0.31% )
24.644 +- 0.071 seconds time elapsed ( +- 0.29% )
# cpufreq-max:
24.452 +- 0.110 seconds time elapsed ( +- 0.45% )
24.482 +- 0.048 seconds time elapsed ( +- 0.20% )
# v6.7.0+sched/core:
# bootup default schedutil governor:
24.666 +- 0.063 seconds time elapsed ( +- 0.26% )
24.809 +- 0.118 seconds time elapsed ( +- 0.48% )
The fully-cached build numbers are very close to each other, and during the
hot phase of the kernel build all CPUs are saturated.
The 2x performance regression that Linus is seeing is either some
pathological wakeup behavior, or perhaps the cores don't transition
frequencies? The difference between the lowest and highest frequency is
pretty substantial (at least on my box):
cpu MHz : 2200.000
...
cpu MHz : 4000.000
There was *one* test when the tree was cache-cold, when I saw really bad
performance (which I didn't really expect with my nvram system), with -j32
builds:
Performance counter stats for 'make -j32 kernel/' (3 runs):
64.34 +- 39.22 seconds time elapsed ( +- 60.95% )
25.08 +- 0.142 seconds time elapsed ( +- 0.56% )
24.97 +- 0.072 seconds time elapsed ( +- 0.29% )
Unfortunately that outlier was on a vanilla v6.7 bootup.
As a next step I could try Linus's specific config, maybe there's some
detail in it that makes the difference.
The commit itself that Linus bisected to (9c0b4bb7f6303c) doesn't *seem*
wrong in itself, especially without uclamp [I presume Linus doesn't use
CONFIG_UCLAMP_TASK=y and the cpu.uclamp.min/uclamp.max cgroup interface
that goes with it?], but the commit changes how we use sched_util metrics,
which could change scheduling patterns - which is why I was spending many
hours yesterday and today trying to find a pathological workload to
reproduce this. No luck so far.
Linus: I can send a pull request for the 2-commit revert, or maybe you
could try Vincent's guess-patch that tries to restore to previous behavior
as closely as possible.
Thanks,
Ingo
next prev parent reply other threads:[~2024-01-12 18:24 UTC|newest]
Thread overview: 54+ messages / expand[flat|nested] mbox.gz Atom feed top
2023-10-28 12:23 [GIT PULL] Scheduler changes for v6.7 Ingo Molnar
2023-10-30 23:50 ` pr-tracker-bot
2024-01-08 14:07 ` [GIT PULL] Scheduler changes for v6.8 Ingo Molnar
2024-01-09 4:04 ` pr-tracker-bot
2024-01-10 22:19 ` Linus Torvalds
2024-01-10 22:41 ` Linus Torvalds
2024-01-10 22:57 ` Linus Torvalds
2024-01-11 8:11 ` Vincent Guittot
2024-01-11 17:45 ` Linus Torvalds
2024-01-11 17:53 ` Linus Torvalds
2024-01-11 18:16 ` Vincent Guittot
2024-01-12 14:23 ` Dietmar Eggemann
2024-01-12 16:58 ` Vincent Guittot
2024-01-12 18:18 ` Qais Yousef
2024-01-12 19:03 ` Vincent Guittot
2024-01-12 20:30 ` Linus Torvalds
2024-01-12 20:49 ` Linus Torvalds
2024-01-12 21:04 ` Linus Torvalds
2024-01-13 1:04 ` Qais Yousef
2024-01-13 1:24 ` Linus Torvalds
2024-01-13 1:31 ` Linus Torvalds
2024-01-13 10:47 ` Vincent Guittot
2024-01-13 18:33 ` Qais Yousef
2024-01-13 18:37 ` Qais Yousef
2024-01-11 11:09 ` [GIT PULL] scheduler fixes Ingo Molnar
2024-01-11 13:04 ` Vincent Guittot
2024-01-11 20:48 ` [PATCH] Revert "sched/cpufreq: Rework schedutil governor performance estimation" and dependent commit Ingo Molnar
2024-01-11 22:22 ` Vincent Guittot
2024-01-12 18:24 ` Ingo Molnar [this message]
2024-01-12 18:26 ` [GIT PULL] Scheduler changes for v6.8 Ingo Molnar
2024-01-14 9:12 ` Wyes Karny
2024-01-14 11:18 ` Vincent Guittot
2024-01-14 12:37 ` Wyes Karny
2024-01-14 13:02 ` Dietmar Eggemann
2024-01-14 13:05 ` Vincent Guittot
2024-01-14 13:03 ` Vincent Guittot
2024-01-14 15:12 ` Qais Yousef
2024-01-14 15:20 ` Vincent Guittot
2024-01-14 19:58 ` Qais Yousef
2024-01-14 23:37 ` Qais Yousef
2024-01-15 6:25 ` Wyes Karny
2024-01-15 11:59 ` Qais Yousef
2024-01-15 8:21 ` Vincent Guittot
2024-01-15 12:09 ` Qais Yousef
2024-01-15 13:26 ` Vincent Guittot
2024-01-15 14:03 ` Dietmar Eggemann
2024-01-15 15:26 ` Vincent Guittot
2024-01-15 20:05 ` Dietmar Eggemann
2024-01-15 8:42 ` David Laight
2024-01-14 18:11 ` Wyes Karny
2024-01-14 18:18 ` Vincent Guittot
2024-01-11 9:33 ` Ingo Molnar
2024-01-11 11:14 ` [tip: sched/urgent] Revert "sched/cpufreq: Rework schedutil governor performance estimation" and dependent commits tip-bot2 for Ingo Molnar
2024-01-11 20:55 ` [tip: sched/urgent] Revert "sched/cpufreq: Rework schedutil governor performance estimation" and dependent commit tip-bot2 for Ingo Molnar
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=ZaGDziEnKJ988zHh@gmail.com \
--to=mingo@kernel.org \
--cc=bristot@redhat.com \
--cc=bsegall@google.com \
--cc=dietmar.eggemann@arm.com \
--cc=juri.lelli@redhat.com \
--cc=linux-kernel@vger.kernel.org \
--cc=mgorman@suse.de \
--cc=peterz@infradead.org \
--cc=rostedt@goodmis.org \
--cc=tglx@linutronix.de \
--cc=torvalds@linux-foundation.org \
--cc=vincent.guittot@linaro.org \
--cc=vschneid@redhat.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.