From: Ingo Molnar <mingo@kernel.org>
To: Arnaldo Carvalho de Melo <acme@ghostprotocols.net>
Cc: Linus Torvalds <torvalds@linux-foundation.org>,
David Ahern <dsahern@gmail.com>,
Linux Kernel Mailing List <linux-kernel@vger.kernel.org>,
Peter Zijlstra <a.p.zijlstra@chello.nl>,
Thomas Gleixner <tglx@linutronix.de>,
Andrew Morton <akpm@linux-foundation.org>,
Jiri Olsa <jolsa@redhat.com>, Namhyung Kim <namhyung@gmail.com>
Subject: [PATCH] perf autodep: Speed up the 'all features are present' case
Date: Tue, 1 Oct 2013 14:51:16 +0200 [thread overview]
Message-ID: <20131001125116.GA19606@gmail.com> (raw)
In-Reply-To: <20131001120455.GA8001@gmail.com>
* Ingo Molnar <mingo@kernel.org> wrote:
> And the actual feature check is roughly 0.330 msecs of that:
>
> comet:~/tip/tools/perf/config/feature-checks> time ( make -j >/dev/null; \
> for N in stackprotector-all volatile-register-var fortify-source libelf \
> libelf-mmap glibc dwarf libelf-getphdrnum libunwind libaudit libslang gtk2 \
> gtk2-infobar libperl libpython libpython-version libbfd on-exit backtrace \
> libnuma; do make test-$N >/dev/null; done )
>
> real 0m0.330s
> user 0m0.290s
> sys 0m0.031s
>
> With 0.150 msecs spent elsewhere.
>
> So there's more speedups possible I think, for example we could
> construct an 'optimistic' testcase that is generated live and includes a
> concatenation of all the testcases.
>
> If the build of that file succeeds then we have a really efficient
> fast-path both in the first-build and in the repeat-build case.
>
> If that build fails then we do the more finegrained feature check.
>
> Thoughts?
So, something like the patch below. It contains test-all.c "testcase from
hell" which will succeed on a well configured system.
With this final trick I got a ridiculous speedup in auto-detection speed,
for the rebuild-again case:
comet:~/tip/tools/perf> perf stat --null --repeat 5 make Makefile
Auto-detecting system features:
... stackprotector-all: [ on ]
... volatile-register-var: [ on ]
... fortify-source: [ on ]
... libelf: [ on ]
... libelf-mmap: [ on ]
... glibc: [ on ]
... dwarf: [ on ]
... libelf-getphdrnum: [ on ]
... libunwind: [ on ]
... libaudit: [ on ]
... libslang: [ on ]
... gtk2: [ on ]
... gtk2-infobar: [ on ]
... libperl: [ on ]
... libpython: [ on ]
... libpython-version: [ on ]
... libbfd: [ on ]
... on-exit: [ on ]
... backtrace: [ on ]
... libnuma: [ on ]
make: Nothing to be done for `Makefile'.
Performance counter stats for 'make Makefile' (5 runs):
0.183843005 seconds time elapsed ( +- 0.21% )
So the cached build time is down from 3.300 secs to 0.183 secs, an 18-fold
speedup.
It's still full auto-detection, because the -MD dependency generation
works for test-all.c as well. For example once I remove the
'perl-ExtUtils-Embed' package, I immediately get the correct, adapted
auto-dep output with the 'libperl' testcase showing 'OFF':
comet:~/tip/tools/perf> make Makefile
Auto-detecting system features:
... stackprotector-all: [ on ]
... volatile-register-var: [ on ]
... fortify-source: [ on ]
... libelf: [ on ]
... libelf-mmap: [ on ]
... glibc: [ on ]
... dwarf: [ on ]
... libelf-getphdrnum: [ on ]
... libunwind: [ on ]
... libaudit: [ on ]
... libslang: [ on ]
... gtk2: [ on ]
... gtk2-infobar: [ on ]
... libperl: [ OFF ]
... libpython: [ on ]
... libpython-version: [ on ]
... libbfd: [ on ]
... on-exit: [ on ]
... backtrace: [ on ]
... libnuma: [ on ]
Note that at this point this is just a proof-of-concept patch for
performance testing, nothing mergable yet.
In particular I'm not happy yet with the current construction of
test-all.c and its build method: it's the result of concatenation of files
and build arguments plus further massaging.
That should probably be automated in some fashion, to make it easier to
add new testcases and to make the whole construct more maintainable.
Thanks,
Ingo
=========================>
Subject: perf autodep: Speed up the 'all features are present' case
From: Ingo Molnar <mingo@kernel.org>
Date: Tue Oct 1 14:14:31 CEST 2013
---
tools/perf/config/Makefile | 18 ++
tools/perf/config/feature-checks/Makefile | 3
tools/perf/config/feature-checks/test-all.c | 196 ++++++++++++++++++++++++++++
3 files changed, 215 insertions(+), 2 deletions(-)
Index: tip/tools/perf/config/Makefile
===================================================================
--- tip.orig/tools/perf/config/Makefile
+++ tip/tools/perf/config/Makefile
@@ -94,16 +94,30 @@ define feature_check_code
feature-$(1) := $(shell make -C config/feature-checks test-$1 >/dev/null 2>/dev/null && echo 1 || echo 0)
endef
+feature_set = $(eval $(feature_set_code))
+define feature_set_code
+ feature-$(1) := 1
+endef
+
#
# Build the feature check binaries in parallel, ignore errors, ignore return value and suppress output:
#
$(info )
$(info Auto-detecting system features:)
-$(shell make -i -j -C config/feature-checks >/dev/null 2>&1)
FEATURE_TESTS = stackprotector-all volatile-register-var fortify-source libelf libelf-mmap glibc dwarf libelf-getphdrnum libunwind libaudit libslang gtk2 gtk2-infobar libperl libpython libpython-version libbfd on-exit backtrace libnuma
-$(foreach feat,$(FEATURE_TESTS),$(call feature_check,$(feat)))
+#
+# Special fast-path for the 'all features are available' case:
+#
+$(call feature_check,all)
+
+ifeq ($(feature-all), 1)
+ $(foreach feat,$(FEATURE_TESTS),$(call feature_set,$(feat)))
+else
+ $(shell make -i -j -C config/feature-checks >/dev/null 2>&1)
+ $(foreach feat,$(FEATURE_TESTS),$(call feature_check,$(feat)))
+endif
feature_print = $(eval $(feature_print_code))
Index: tip/tools/perf/config/feature-checks/Makefile
===================================================================
--- tip.orig/tools/perf/config/feature-checks/Makefile
+++ tip/tools/perf/config/feature-checks/Makefile
@@ -99,6 +99,9 @@ test-on-exit: test-on-exit.c
test-backtrace: test-backtrace.c
$(CC) -o $@ $@.c
+test-all:
+ $(CC) -o $@ $@.c -Werror -fstack-protector -Wvolatile-register-var -O2 -Werror -D_FORTIFY_SOURCE=2 -ldw -lelf -lnuma -lunwind -lunwind-x86_64 -lelf -laudit -I/usr/include/slang -lslang $(shell pkg-config --libs --cflags gtk+-2.0 2>/dev/null) $(FLAGS_PERL_EMBED) $(FLAGS_PYTHON_EMBED) -DPACKAGE='perf' -DPACKAGE=perf -lbfd -ldl
+
-include *.d */*.d
###############################
Index: tip/tools/perf/config/feature-checks/test-all.c
===================================================================
--- /dev/null
+++ tip/tools/perf/config/feature-checks/test-all.c
@@ -0,0 +1,196 @@
+
+#pragma GCC diagnostic ignored "-Wstrict-prototypes"
+
+#include <Python.h>
+
+#include <EXTERN.h>
+#include <perl.h>
+
+#include <stdio.h>
+#include <libelf.h>
+#include <gnu/libc-version.h>
+#include <dwarf.h>
+#include <elfutils/libdw.h>
+#include <elfutils/version.h>
+#include <libelf.h>
+#include <libunwind.h>
+#include <stdlib.h>
+#include <libaudit.h>
+#include <slang.h>
+#include <gtk/gtk.h>
+#include <bfd.h>
+#include <stdio.h>
+#include <execinfo.h>
+#include <stdio.h>
+#include <numa.h>
+#include <numaif.h>
+
+#pragma GCC diagnostic error "-Wstrict-prototypes"
+
+int main1(void)
+{
+ return puts("hi");
+}
+
+int main2(void)
+{
+ return puts("hi");
+}
+
+int main3(void)
+{
+ return puts("hi");
+}
+
+int main4(void)
+{
+ Elf *elf = elf_begin(0, ELF_C_READ, 0);
+ return (long)elf;
+}
+#
+int main5(void)
+{
+ Elf *elf = elf_begin(0, ELF_C_READ_MMAP, 0);
+ return (long)elf;
+}
+
+int main6(void)
+{
+ const char *version = gnu_get_libc_version();
+ return (long)version;
+}
+
+int main7(void)
+{
+ Dwarf *dbg = dwarf_begin(0, DWARF_C_READ);
+ return (long)dbg;
+}
+
+int main8(void)
+{
+ size_t dst;
+ return elf_getphdrnum(0, &dst);
+}
+
+extern int UNW_OBJ(dwarf_search_unwind_table) (unw_addr_space_t as,
+ unw_word_t ip,
+ unw_dyn_info_t *di,
+ unw_proc_info_t *pi,
+ int need_unwind_info, void *arg);
+
+
+#define dwarf_search_unwind_table UNW_OBJ(dwarf_search_unwind_table)
+
+int main9(void)
+{
+ unw_addr_space_t addr_space;
+ addr_space = unw_create_addr_space(NULL, 0);
+ unw_init_remote(NULL, addr_space, NULL);
+ dwarf_search_unwind_table(addr_space, 0, NULL, NULL, 0, NULL);
+ return 0;
+}
+
+int main10(void)
+{
+ printf("error message: %s\n", audit_errno_to_name(0));
+ return audit_open();
+}
+
+int main11(void)
+{
+ return SLsmg_init_smg();
+}
+
+int main12(int argc, char *argv[])
+{
+ gtk_init(&argc, &argv);
+
+ return 0;
+}
+
+int main13(void)
+{
+ gtk_info_bar_new();
+
+ return 0;
+}
+
+int main14(void)
+{
+ perl_alloc();
+
+ return 0;
+}
+
+int main15(void)
+{
+ Py_Initialize();
+ return 0;
+}
+
+#if PY_VERSION_HEX >= 0x03000000
+ #error
+#endif
+
+int main16(void)
+{
+ return 0;
+}
+
+int main17(void)
+{
+ bfd_demangle(0, 0, 0);
+ return 0;
+}
+
+void exit_function(int x, void *y)
+{
+}
+
+int main18(void)
+{
+ return on_exit(exit_function, NULL);
+}
+
+int main19(void)
+{
+ void *backtrace_fns[1];
+ size_t entries;
+
+ entries = backtrace(backtrace_fns, 1);
+ backtrace_symbols(backtrace_fns, entries);
+
+ return 0;
+}
+
+int main20(void)
+{
+ numa_available();
+ return 0;
+}
+
+int main(int argc, char *argv[])
+{
+ main1();
+ main2();
+ main3();
+ main4();
+ main5();
+ main6();
+ main7();
+ main8();
+ main9();
+ main10();
+ main11();
+ main12(argc, argv);
+ main13();
+ main14();
+ main15();
+ main16();
+ main17();
+ main18();
+ main19();
+ main20();
+
+ return 0;
+}
next prev parent reply other threads:[~2013-10-01 12:51 UTC|newest]
Thread overview: 60+ messages / expand[flat|nested] mbox.gz Atom feed top
2013-09-12 13:38 [GIT PULL] perf fixes Ingo Molnar
2013-09-12 18:03 ` Linus Torvalds
2013-09-12 18:10 ` Linus Torvalds
2013-09-12 18:43 ` Arnaldo Carvalho de Melo
2013-09-12 19:12 ` Arnaldo Carvalho de Melo
2013-09-12 19:13 ` Linus Torvalds
2013-09-12 19:55 ` Ingo Molnar
2013-09-12 19:58 ` David Ahern
2013-09-12 20:02 ` Arnaldo Carvalho de Melo
2013-09-12 20:31 ` Ingo Molnar
2013-09-12 20:43 ` Ingo Molnar
2013-09-15 9:10 ` [PATCH] perf test-hack: Split out feature tests to cache them and to build them in parallel Ingo Molnar
2013-09-30 16:42 ` [PATCH] perf auto-dep: Speed up feature tests by building " Ingo Molnar
2013-09-30 17:12 ` Arnaldo Carvalho de Melo
2013-09-30 17:27 ` Arnaldo Carvalho de Melo
2013-09-30 17:30 ` Arnaldo Carvalho de Melo
2013-09-30 17:36 ` Arnaldo Carvalho de Melo
2013-09-30 17:39 ` Arnaldo Carvalho de Melo
2013-09-30 17:46 ` Arnaldo Carvalho de Melo
2013-09-30 18:02 ` Arnaldo Carvalho de Melo
2013-09-30 19:15 ` Ingo Molnar
2013-09-30 19:09 ` Ingo Molnar
2013-09-30 17:34 ` Linus Torvalds
2013-09-30 17:53 ` Arnaldo Carvalho de Melo
2013-09-30 19:04 ` Ingo Molnar
2013-10-01 11:34 ` [PATCH] perf autodep: Remove strlcpy feature check, add __weak strlcpy implementation Ingo Molnar
2013-10-01 12:04 ` Ingo Molnar
2013-10-01 12:48 ` Arnaldo Carvalho de Melo
2013-10-01 12:51 ` Ingo Molnar [this message]
2013-10-01 14:46 ` [PATCH] perf tools: Speed up git-version test on re-make Ingo Molnar
2013-10-02 6:47 ` Namhyung Kim
2013-10-02 6:50 ` Ingo Molnar
2013-10-02 8:04 ` Namhyung Kim
2013-10-01 15:27 ` [PATCH] perf autodep: Remove strlcpy feature check, add __weak strlcpy implementation Ingo Molnar
2013-10-01 15:29 ` [PATCH] perf tools: Speed up the final link Ingo Molnar
2013-10-01 7:04 ` [PATCH] perf auto-dep: Speed up feature tests by building them in parallel Geert Uytterhoeven
2013-10-01 8:38 ` Ingo Molnar
2013-10-02 6:05 ` Namhyung Kim
2013-10-02 6:28 ` Ingo Molnar
2013-10-02 9:26 ` Jiri Olsa
2013-10-02 10:11 ` Ingo Molnar
2013-09-12 20:18 ` [GIT PULL] perf fixes Ingo Molnar
2013-09-12 20:38 ` Arnaldo Carvalho de Melo
2013-09-12 20:46 ` Ingo Molnar
2013-09-12 21:09 ` David Ahern
2013-09-12 21:18 ` Ingo Molnar
2013-09-12 22:10 ` David Ahern
2013-09-13 5:09 ` Ingo Molnar
2013-09-13 9:32 ` Jean Pihet
2013-09-13 9:45 ` Ingo Molnar
2013-09-13 17:15 ` Jean Pihet
2013-09-12 18:51 ` Linus Torvalds
2013-09-12 20:33 ` Ingo Molnar
2013-09-12 20:38 ` Linus Torvalds
2013-09-12 20:49 ` Ingo Molnar
2013-09-12 20:52 ` Linus Torvalds
2013-09-12 21:01 ` Ingo Molnar
2013-09-12 20:10 ` Ingo Molnar
2013-10-02 7:31 ` [PATCH] tools/perf: Fix double/triple-build of the feature detection logic during 'make install' et al Ingo Molnar
2013-10-02 9:28 ` [PATCH] tools/perf/build: Automatically build in parallel, based on number of CPUs in the system Ingo Molnar
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20131001125116.GA19606@gmail.com \
--to=mingo@kernel.org \
--cc=a.p.zijlstra@chello.nl \
--cc=acme@ghostprotocols.net \
--cc=akpm@linux-foundation.org \
--cc=dsahern@gmail.com \
--cc=jolsa@redhat.com \
--cc=linux-kernel@vger.kernel.org \
--cc=namhyung@gmail.com \
--cc=tglx@linutronix.de \
--cc=torvalds@linux-foundation.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.