From: Joel Fernandes <joel@joelfernandes.org>
To: Kees Cook <keescook@chromium.org>
Cc: "Zhengyejian (Zetta)" <zhengyejian1@huawei.com>,
rostedt@goodmis.org, ccross@android.com,
linux-kernel@vger.kernel.org,
Zhangjinhao <zhangjinhao2@huawei.com>
Subject: Re: [BUG] I found a bug when try to enable record_ftrace
Date: Tue, 8 Jun 2021 15:16:44 -0400 [thread overview]
Message-ID: <YL/CHEbw5L2Selfm@google.com> (raw)
In-Reply-To: <202106021042.8527F3AB30@keescook>
On Wed, Jun 02, 2021 at 10:43:29AM -0700, Kees Cook wrote:
>
> On Wed, Jun 02, 2021 at 03:42:23PM +0800, Zhengyejian (Zetta) wrote:
> > Hello,
> >
> > There may be a deadlock caused by ftrace recursion when try to enable
> > record_ftrace.
>
> Hi,
>
> Thanks for the report. Joel, is this something you can take a look at?
Kees,
Sorry for late reply. Right now I am dealing with a production-stop bug and
would not likely get time. Plus I feel a bit less motivated, because the
ftrace in pstore the way it stands now is unusable due to perf issues anyway
(I wonder if anyone uses it at all now). I would rather someone who has time
to work on this try to revive: https://lkml.org/lkml/2020/9/2/1075 . Should
we just delete pstore function tracing till then?
-Joel
> -Kees
>
> > I'd like to known if the patchset
> > (https://lore.kernel.org/lkml/20201106023235.367190737@goodmis.org/) is
> > trying to fix it.
> >
> > Procedure for reproducing the problem is:
> > 1) this problem appears both in 5.13-rc4 and 5.10;
> >
> > 2) my work environment is:
> > qemu-arm version 4.0.0
> > arm-linux-gnueabi-gcc (Ubuntu/Linaro 7.5.0-3ubuntu1~18.04) 7.5.0
> >
> > 3) then try to enable record_ftrace:
> > / # mount -t pstore pstore /sys/fs/pstore
> > / # echo 1 > /sys/kernel/debug/pstore/record_ftrace
> >
> > 4) then system appears to be stuck, and use arm-linux-gnueabi-gdb dump the
> > following call stack:
> > #0 arch_spin_lock (lock=0x811a0f98) at
> > /home/zyj/Linux/linux-master/arch/arm/include/asm/spinlock.h:74
> > #1 do_raw_spin_lock_flags (flags=<synthetic pointer>, lock=0x811a0f98)
> > at /home/zyj/Linux/linux-master/include/linux/spinlock.h:195
> > #2 __raw_spin_lock_irqsave (lock=0x811a0f98) at
> > /home/zyj/Linux/linux-master/include/linux/spinlock_api_smp.h:119
> > #3 _raw_spin_lock_irqsave (lock=lock@entry=0x811a0f98) at
> > /home/zyj/Linux/linux-master/kernel/locking/spinlock.c:159
> > #4 0x8046c6e0 in buffer_size_add (prz=prz@entry=0x811a0f80,
> > a=a@entry=16) at /home/zyj/Linux/linux-master/fs/pstore/ram_core.c:82
> > #5 0x8046cc20 in persistent_ram_write (prz=0x811a0f80, s=0x81137b80,
> > count=16) at /home/zyj/Linux/linux-master/fs/pstore/ram_core.c:327
> > #6 0x8046b438 in ramoops_pstore_write (record=0x81137b90) at
> > /home/zyj/Linux/linux-master/fs/pstore/ram.c:331
> > #7 0x8046add8 in pstore_ftrace_call (ip=2156609456,
> > parent_ip=2152122068, op=<optimized out>, fregs=<optimized out>)
> > at /home/zyj/Linux/linux-master/fs/pstore/ftrace.c:54
> > #8 0x801dd580 in __ftrace_ops_list_func (ignored=0x0, fregs=0x0,
> > parent_ip=2152122068, ip=2156609456) at
> > /home/zyj/Linux/linux-master/kernel/trace/ftrace.c:7003
> > #9 ftrace_ops_list_func (ip=2156609456, parent_ip=2152122068,
> > op=<optimized out>, fregs=0x0) at
> > /home/zyj/Linux/linux-master/kernel/trace/ftrace.c:7028
> > #10 0x801109f8 in ftrace_caller () at
> > /home/zyj/Linux/linux-master/arch/arm/kernel/entry-ftrace.S:224
> > #11 0x808b3fb4 in _raw_spin_unlock_irqrestore
> > (lock=lock@entry=0x811a0f98, flags=1610612883) at
> > /home/zyj/Linux/linux-master/kernel/locking/spinlock.c:190
> > #12 0x8046c6d4 in buffer_size_add (prz=prz@entry=0x811a0f80,
> > a=a@entry=16) at /home/zyj/Linux/linux-master/fs/pstore/ram_core.c:95
> > #13 0x8046cc20 in persistent_ram_write (prz=0x811a0f80, s=0x81137cf8,
> > count=16) at /home/zyj/Linux/linux-master/fs/pstore/ram_core.c:327
> > #14 0x8046b438 in ramoops_pstore_write (record=0x81137d08) at
> > /home/zyj/Linux/linux-master/fs/pstore/ram.c:331
> > #15 0x8046add8 in pstore_ftrace_call (ip=2148632188,
> > parent_ip=2148601660, op=<optimized out>, fregs=<optimized out>)
> > at /home/zyj/Linux/linux-master/fs/pstore/ftrace.c:54
> > #16 0x801dd580 in __ftrace_ops_list_func (ignored=0x0, fregs=0x0,
> > parent_ip=2148601660, ip=2148632188) at
> > /home/zyj/Linux/linux-master/kernel/trace/ftrace.c:7003
> > #17 ftrace_ops_list_func (ip=2148632188, parent_ip=2148601660,
> > op=<optimized out>, fregs=0x0) at
> > /home/zyj/Linux/linux-master/kernel/trace/ftrace.c:7028
> > #18 0x801109f8 in ftrace_caller () at
> > /home/zyj/Linux/linux-master/arch/arm/kernel/entry-ftrace.S:224
> > #19 0x80118680 in __set_fixmap (idx=idx@entry=FIX_TEXT_POKE0,
> > phys=phys@entry=0, prot=prot@entry=0) at
> > /home/zyj/Linux/linux-master/arch/arm/mm/mmu.c:385
> > #20 0x80110f3c in patch_unmap (flags=<synthetic pointer>, fixmap=129) at
> > /home/zyj/Linux/linux-master/arch/arm/kernel/patch.c:45
> > #21 __patch_text_real (addr=addr@entry=0x808b3fb0
> > <_raw_spin_unlock_irqrestore+16>, insn=insn@entry=3957420680,
> > remap=remap@entry=true)
> > at /home/zyj/Linux/linux-master/arch/arm/kernel/patch.c:104
> > #22 0x80110b40 in __patch_text (insn=3957420680, addr=0x808b3fb0
> > <_raw_spin_unlock_irqrestore+16>) at
> > /home/zyj/Linux/linux-master/arch/arm/include/asm/patch.h:10
> > #23 ftrace_modify_code (pc=2156609456, old=3904716800, new=3957420680,
> > validate=true) at /home/zyj/Linux/linux-master/arch/arm/kernel/ftrace.c:95
> > #24 0x80110cf4 in ftrace_make_call (rec=rec@entry=0x81007ce0,
> > addr=addr@entry=2148600280) at
> > /home/zyj/Linux/linux-master/arch/arm/kernel/ftrace.c:132
> > #25 0x801dda40 in __ftrace_replace_code (rec=rec@entry=0x81007ce0,
> > enable=enable@entry=true) at
> > /home/zyj/Linux/linux-master/kernel/trace/ftrace.c:2524
> > #26 0x801de17c in ftrace_replace_code (mod_flags=mod_flags@entry=1) at
> > /home/zyj/Linux/linux-master/kernel/trace/ftrace.c:2554
> > #27 0x801de39c in ftrace_modify_all_code (command=5) at
> > /home/zyj/Linux/linux-master/kernel/trace/ftrace.c:2698
> > #28 0x80110ad8 in __ftrace_modify_code (data=<optimized out>) at
> > /home/zyj/Linux/linux-master/arch/arm/kernel/ftrace.c:39
> > #29 0x801d4da4 in multi_cpu_stop (data=data@entry=0x81c69d84) at
> > /home/zyj/Linux/linux-master/kernel/stop_machine.c:240
> > #30 0x801d4a14 in cpu_stopper_thread (cpu=<optimized out>) at
> > /home/zyj/Linux/linux-master/kernel/stop_machine.c:511
> > #31 0x8014de3c in smpboot_thread_fn (data=0x810c1c80) at
> > /home/zyj/Linux/linux-master/kernel/smpboot.c:165
> > #32 0x8014a0dc in kthread (_create=0x810c1d40) at
> > /home/zyj/Linux/linux-master/kernel/kthread.c:313
> > #33 0x80100150 in ret_from_fork () at
> > /home/zyj/Linux/linux-master/arch/arm/kernel/entry-common.S:158
> > Backtrace stopped: previous frame identical to this frame (corrupt
> > stack?)
> >
> > See above #7~#15, there is a recursion in function pstore_ftrace_call(), and
> > a spin_lock(lock=0x811a0f98) is hold since unlock operation in above #11 not
> > finished. Then in above #0, trying to acquire same lock cause a deadlock.
> >
> > Enabling 'record_ftrace' seems a basic operation of pstore/ftrace, Does it
> > mean this feature is not available for a while?
> >
> > Best Regards,
> > Zheng
>
> --
> Kees Cook
next prev parent reply other threads:[~2021-06-08 19:34 UTC|newest]
Thread overview: 6+ messages / expand[flat|nested] mbox.gz Atom feed top
2021-06-02 7:42 [BUG] I found a bug when try to enable record_ftrace Zhengyejian (Zetta)
2021-06-02 13:52 ` Steven Rostedt
2021-06-02 17:43 ` Kees Cook
2021-06-08 19:16 ` Joel Fernandes [this message]
2021-06-09 18:12 ` Kees Cook
2021-06-10 17:26 ` Steven Rostedt
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=YL/CHEbw5L2Selfm@google.com \
--to=joel@joelfernandes.org \
--cc=ccross@android.com \
--cc=keescook@chromium.org \
--cc=linux-kernel@vger.kernel.org \
--cc=rostedt@goodmis.org \
--cc=zhangjinhao2@huawei.com \
--cc=zhengyejian1@huawei.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.