From: Xunlei Pang <xlpang-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>
To: Harald Hoyer <harald-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>
Cc: initramfs-u79uwXL29TY76Z2rM5mHXA@public.gmane.org,
Dave Young <dyoung-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>,
Pratyush Anand <panand-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>,
Xunlei Pang <xlpang-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>
Subject: [PATCH v3 1/3] 99base: add memtrace-ko.sh to debug kernel module large memory consumption
Date: Mon, 14 Nov 2016 14:11:24 +0800 [thread overview]
Message-ID: <1479103886-24568-1-git-send-email-xlpang@redhat.com> (raw)
The current method for memory debug is to use "rd.memdebug=[0-3]",
it is not enough for debugging kernel modules. For example, when we
want to find out which kernel module consumes a large amount of memory,
"rd.memdebug=[0-3]" won't help too much.
A better way is needed to achieve this requirement, this is useful for
kdump OOM debugging.
The principle of this patch is to use kernel trace to track slab and
buddy allocation calls during kernel module loading(module_init), thus
we can analyze all the trace data and get the total memory consumption.
As for large slab allocation, it will probably fall into buddy allocation,
thus tracing "mm_page_alloc" alone should be enough for the purpose(this
saves quite some trace buffer memory, also large free is quite unlikey
during module loading, we neglect those memory free events).
The trace events include memory calls under "tracing/events/":
kmem/mm_page_alloc
We also inpect the following events to detect the module loading:
module/module_load
module/module_put
Since we use filters to trace events, the final trace data size won't
be too big. Users can adjust the trace buffer size via "trace_buf_size"
kernel boot command line as needed.
We can get the module name and task pid from "module_load" event which
also mark the beginning of the loading, and module_put called by the
same task pid implies the end of the loading. So the memory events
recorded in between by the same task pid are consumed by this module
during loading(i.e. modprobe or module_init()).
With these information, we can record the rough total memory(the larger,
the more precise the result will be) consumption involved by each kernel
module loading.
Thus we introduce this shell script to find out which kernel module
consumes a large amount of memory during loading. Use "rd.memdebug=4"
as the tigger.
After applying this patch and specifying "rd.memdebug=4", during booting
it will print out something extra like below:
0 pages consumed by "pata_acpi"
0 pages consumed by "ata_generic"
1 pages consumed by "drm"
0 pages consumed by "ttm"
0 pages consumed by "drm_kms_helper"
835 pages consumed by "qxl"
0 pages consumed by "mii"
6 pages consumed by "8139cp"
0 pages consumed by "virtio"
0 pages consumed by "virtio_ring"
9 pages consumed by "virtio_pci"
1 pages consumed by "8139too"
0 pages consumed by "serio_raw"
0 pages consumed by "crc32c_intel"
199 pages consumed by "virtio_console"
0 pages consumed by "libcrc32c"
9 pages consumed by "xfs"
From the print, we see clearly that "qxl" consumed the most memory.
This file will be installed as a separate executable named "tracekomem"
in the following patch.
Signed-off-by: Xunlei Pang <xlpang-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>
---
Series change from v2:
Improved documentation, no functional change.
modules.d/99base/memtrace-ko.sh | 169 ++++++++++++++++++++++++++++++++++++++++
1 file changed, 169 insertions(+)
create mode 100755 modules.d/99base/memtrace-ko.sh
diff --git a/modules.d/99base/memtrace-ko.sh b/modules.d/99base/memtrace-ko.sh
new file mode 100755
index 0000000..f4720bf
--- /dev/null
+++ b/modules.d/99base/memtrace-ko.sh
@@ -0,0 +1,169 @@
+# Try to find out kernel modules with large total memory allocation during loading.
+# For large slab allocation, it will fall into buddy, thus tracing "mm_page_alloc"
+# alone should be enough for the purpose.
+
+# "sys/kernel/tracing" has the priority if exists.
+get_trace_base() {
+ # trace access through debugfs would be obsolete if "/sys/kernel/tracing" is available.
+ if [[ -d "/sys/kernel/tracing" ]]; then
+ echo "/sys/kernel"
+ else
+ echo "/sys/kernel/debug"
+ fi
+}
+
+# We want to enable these trace events.
+get_want_events() {
+ echo "module:module_put module:module_load kmem:mm_page_alloc"
+}
+
+get_event_filter() {
+ echo "comm == systemd-udevd || comm == modprobe || comm == insmod"
+}
+
+is_trace_ready() {
+ local trace_base want_events current_events
+
+ trace_base=$(get_trace_base)
+ ! [[ -f "$trace_base/tracing/trace" ]] && return 1
+
+ [[ $(cat $trace_base/tracing/tracing_on) = 0 ]] && return 1
+
+ # Also check if trace events were properly setup.
+ want_events=$(get_want_events)
+ current_events=$(echo $(cat $trace_base/tracing/set_event))
+ [[ "$current_events" != "$want_events" ]] && return 1
+
+ return 0
+}
+
+prepare_trace() {
+ local trace_base
+
+ trace_base=$(get_trace_base)
+ # old debugfs interface case.
+ if ! [[ -d "$trace_base/tracing" ]]; then
+ mount none -t debugfs $trace_base
+ # new tracefs interface case.
+ elif ! [[ -f "$trace_base/tracing/trace" ]]; then
+ mount none -t tracefs "$trace_base/tracing"
+ fi
+
+ if ! [[ -f "$trace_base/tracing/trace" ]]; then
+ echo "WARN: Mount trace failed for kernel module memory analyzing."
+ return
+ fi
+
+ # Active all the wanted trace events.
+ echo "$(get_want_events)" > $trace_base/tracing/set_event
+
+ # There are three kinds of known applications for module loading:
+ # "systemd-udevd", "modprobe" and "insmod".
+ # Set them as the global events filter.
+ # NOTE: Some kernel may not support this format of filter, anyway
+ # the operation will fail and it doesn't matter.
+ echo "$(get_event_filter)" > $trace_base/tracing/events/kmem/filter
+ echo "$(get_event_filter)" > $trace_base/tracing/events/module/filter
+
+ # Set the number of comm-pid if supported.
+ if [[ -f "$trace_base/tracing/saved_cmdlines_size" ]]; then
+ # Thanks to filters, 4096 is big enough(also well supported).
+ echo 4096 > $trace_base/tracing/saved_cmdlines_size
+ fi
+
+ # Enable and clear trace data for the first time.
+ echo 1 > $trace_base/tracing/tracing_on
+ echo > $trace_base/tracing/trace
+ echo "Prepare trace success."
+}
+
+parse_trace_data() {
+ local module_name
+ # Indexed by task pid.
+ local -A current_module
+ # Indexed by module name.
+ local -A module_loaded
+ local -A nr_alloc_pages
+
+ cat "$(get_trace_base)/tracing/trace" | while read pid cpu flags ts function
+ do
+ # Skip comment lines
+ if [[ $pid = "#" ]]; then
+ continue
+ fi
+
+ if [[ $function = module_load* ]]; then
+ # One module is being loaded, save the task pid for tracking.
+ module_name=${function#*: }
+ # Remove the trailing after whitespace, there may be the module flags.
+ module_name=${module_name%% *}
+ # Mark current_module to track the task.
+ current_module[$pid]="$module_name"
+ [[ ${module_loaded[$module_name]} ]] && echo "WARN: \"$module_name\" was loaded multiple times!"
+ unset module_loaded[$module_name]
+ nr_alloc_pages[$module_name]=0
+ continue
+ fi
+
+ if ! [[ ${current_module[$pid]} ]]; then
+ continue
+ fi
+
+ # Once we get here, the task is being tracked(is loading a module).
+ # Get the module name.
+ module_name=${current_module[$pid]}
+
+ if [[ $function = module_put* ]]; then
+ # Mark the module as loaded when the first module_put event happens after module_load.
+ echo "${nr_alloc_pages[$module_name]} pages consumed by \"$module_name\""
+ module_loaded[$module_name]=1
+ # Module loading finished, so untrack the task.
+ unset current_module[$pid]
+ continue
+ fi
+
+ if [[ $function = mm_page_alloc* ]]; then
+ order=$(echo $function | sed -e 's/.*order=\([0-9]*\) .*/\1/')
+ nr_alloc_pages[$module_name]=$((${nr_alloc_pages[$module_name]}+$((2 ** $order))))
+ fi
+ done
+}
+
+cleanup_trace() {
+ local trace_base
+
+ if is_trace_ready; then
+ trace_base=$(get_trace_base)
+ echo 0 > $trace_base/tracing/tracing_on
+ echo > $trace_base/tracing/trace
+ echo > $trace_base/tracing/set_event
+ echo 0 > $trace_base/tracing/events/kmem/filter
+ echo 0 > $trace_base/tracing/events/module/filter
+ fi
+}
+
+show_usage() {
+ echo "Find out kernel modules with large memory consumption during loading based on trace."
+ echo "Usage:"
+ echo "1) run it first to setup trace."
+ echo "2) run again to parse the trace data if any."
+ echo "3) run with \"--cleanup\" option to cleanup trace after use."
+}
+
+if [[ $1 = "--help" ]]; then
+ show_usage
+ exit 0
+fi
+
+
+if [[ $1 = "--cleanup" ]]; then
+ cleanup_trace
+ exit 0
+fi
+
+if is_trace_ready ; then
+ echo "tracekomem - Rough memory consumption by loading kernel modules (larger value with better accuracy)"
+ parse_trace_data
+else
+ prepare_trace
+fi
--
1.8.3.1
next reply other threads:[~2016-11-14 6:11 UTC|newest]
Thread overview: 8+ messages / expand[flat|nested] mbox.gz Atom feed top
2016-11-14 6:11 Xunlei Pang [this message]
[not found] ` <1479103886-24568-1-git-send-email-xlpang-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>
2016-11-14 6:11 ` [PATCH v3 2/3] 99base: apply kernel module memory debug support Xunlei Pang
2016-11-14 6:11 ` [PATCH v3 3/3] dracut.cmdline.7.asc: update document for rd.memdebug=4 Xunlei Pang
[not found] ` <1479103886-24568-3-git-send-email-xlpang-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>
2016-11-14 7:03 ` Dracut GitHub Import Bot
2016-11-16 2:30 ` [PATCH v3 1/3] 99base: add memtrace-ko.sh to debug kernel module large memory consumption Dave Young
2016-11-24 2:32 ` Xunlei Pang
[not found] ` <58365135.3040400-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>
2016-11-24 7:39 ` Harald Hoyer
[not found] ` <667da8d4-923d-8cf0-fabe-5f5da7f481e4-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>
2016-11-27 5:45 ` Xunlei Pang
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=1479103886-24568-1-git-send-email-xlpang@redhat.com \
--to=xlpang-h+wxahxf7alqt0dzr+alfa@public.gmane.org \
--cc=dyoung-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org \
--cc=harald-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org \
--cc=initramfs-u79uwXL29TY76Z2rM5mHXA@public.gmane.org \
--cc=panand-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).