From: "josh.nestor" <josh.nestor@protonmail.com>
To: "xenomai@lists.linux.dev" <xenomai@lists.linux.dev>
Subject: Resolving a livelock between Xenomai 3 application and Linux kernel using GDB
Date: Tue, 12 Dec 2023 15:07:13 +0000 [thread overview]
Message-ID: <2eRq8h8p8SWhJ6EnmaVwi0DCGy_pWlWDoNQ5lE_y6bQ09xcwtVzNG5Cg7xc24KX92Dj1nULBDiNT9Gj37p3dHriAYApcqWkmvMVUwP8KjIg=@protonmail.com> (raw)
Dear list,
Apologies if this post appears twice, it looks like it didn't send the first time.
I am trying to resolve a livelock issue I'm having between between the Linux kernel and my Xenomai application.
I am running Xenomai v3.1.1, with Linux 5.4.72, on a Raspberry Pi CM4. The last 2 cores are configured as Xenomai cores. I've enabled debugging symbols on the Linux kernel before compiling. Xenomai is also compiled as part of the kernel rather than a kernel module.
I have attached a JTAG debugger (Segger J-link) to the Raspberry Pi and used OpenOCD connected with 4 instances of GDB, one for each CPU.
From the backtrace it looks like it is stuck at ktime_get() on a sequence lock, probably because somewhere in my application a time-related linux syscall such as gettimeofday() is being called somewhere. I've read that this can lead to livelocks as the reader will have a higher priority than the Linux writing thread.
I'm trying to get a backtrace of Core 2 to try and figure out where this syscall may be called from, however it looks like the debugging symbols are not loaded correctly. I have loaded the debugging symbols for Linux and my application as below, for each of the GDB instances.
file vmlinux
lx-symbols
add-symbol-file my_realtime_application
Are there any other tricks needed to load the debugging symbols for Xenomai and for my realtime application running under Xenomai? And is there a way I can see where each of the realtime threads in my application are currently?
I've also attached a copy of the thread info and backtraces below.
--- Core 0 ---
(gdb) info threads
Id Target Id Frame
* 1 Remote target cpu_relax () at ./arch/arm64/include/asm/processor.h:247
(gdb) bt
#0 cpu_relax () at ./arch/arm64/include/asm/processor.h:247
#1 __read_seqcount_begin (s=<optimized out>) at ./include/linux/seqlock.h:115
#2 raw_read_seqcount_begin (s=<optimized out>) at ./include/linux/seqlock.h:148
#3 read_seqcount_begin (s=<optimized out>) at ./include/linux/seqlock.h:165
#4 ktime_get () at kernel/time/timekeeping.c:757
#5 0xffffffc01014e294 in tick_nohz_irq_enter () at kernel/time/tick-sched.c:1257
#6 tick_irq_enter () at kernel/time/tick-sched.c:1278
#7 0xffffffc0100d3350 in irq_enter () at kernel/softirq.c:354
#8 0xffffffc010122a1c in __handle_domain_irq (domain=0x0, hwirq=15, lookup=false, regs=0xffffff80bf984488)
at kernel/irq/irqdesc.c:671
#9 0xffffffc01009e000 in __ipipe_do_IRQ (irq=<optimized out>, cookie=<optimized out>) at arch/arm64/kernel/ipipe.c:212
#10 0xffffffc0101856c4 in __ipipe_do_sync_stage () at kernel/ipipe/core.c:1702
#11 0xffffffc0101857d0 in __ipipe_sync_stage () at ./include/linux/ipipe.h:116
#12 __ipipe_do_sync_pipeline (top=0xffffffc010cfd400 <ipipe_root>) at kernel/ipipe/core.c:977
#13 0xffffffc010185c80 in __ipipe_sync_pipeline (top=<optimized out>) at ./include/linux/ipipe.h:332
#14 __ipipe_dispatch_irq (irq=1024, flags=<optimized out>) at kernel/ipipe/core.c:1575
#15 0xffffffc0100814bc in __ipipe_grab_irq (irq=<optimized out>, regs=0x400) at arch/arm64/kernel/ipipe.c:202
#16 0xffffffc010081c24 in ipipe_handle_domain_irq (regs=<optimized out>, hwirq=<optimized out>, domain=<optimized out>)
at ./arch/arm64/include/asm/ipipe.h:160
#17 gic_handle_irq (regs=0xffffffc010cfd400 <ipipe_root>) at drivers/irqchip/irq-gic.c:404
#18 0xffffffc010085e04 in handle_arch_irq_pipelined (regs=<optimized out>) at arch/arm64/kernel/irq.c:43
#19 0xffffffc0100834c8 in el1_irq () at arch/arm64/kernel/entry.S:696
Backtrace stopped: previous frame identical to this frame (corrupt stack?)
(gdb)
--- Core 1 ---
(gdb) info threads
Id Target Id Frame
* 1 Remote target cpu_relax () at ./arch/arm64/include/asm/processor.h:247
(gdb) bt
#0 cpu_relax () at ./arch/arm64/include/asm/processor.h:247
#1 __read_seqcount_begin (s=<optimized out>) at ./include/linux/seqlock.h:115
#2 raw_read_seqcount_begin (s=<optimized out>) at ./include/linux/seqlock.h:148
#3 read_seqcount_begin (s=<optimized out>) at ./include/linux/seqlock.h:165
#4 ktime_get () at kernel/time/timekeeping.c:757
#5 0xffffffc01014e294 in tick_nohz_irq_enter () at kernel/time/tick-sched.c:1257
#6 tick_irq_enter () at kernel/time/tick-sched.c:1278
#7 0xffffffc0100d3350 in irq_enter () at kernel/softirq.c:354
#8 0xffffffc010122a1c in __handle_domain_irq (domain=0x0, hwirq=3, lookup=false, regs=0xffffff80bf99f488)
at kernel/irq/irqdesc.c:671
#9 0xffffffc01009e000 in __ipipe_do_IRQ (irq=<optimized out>, cookie=<optimized out>) at arch/arm64/kernel/ipipe.c:212
#10 0xffffffc0101856c4 in __ipipe_do_sync_stage () at kernel/ipipe/core.c:1702
#11 0xffffffc0101857d0 in __ipipe_sync_stage () at ./include/linux/ipipe.h:116
#12 __ipipe_do_sync_pipeline (top=0xffffffc010cfd400 <ipipe_root>) at kernel/ipipe/core.c:977
#13 0xffffffc010185aec in dispatch_irq_head (irq=<optimized out>) at kernel/ipipe/core.c:1454
#14 0xffffffc010185c10 in __ipipe_dispatch_irq (irq=1, flags=<optimized out>) at kernel/ipipe/core.c:1555
#15 0xffffffc0100814bc in __ipipe_grab_irq (irq=<optimized out>, regs=0x100) at arch/arm64/kernel/ipipe.c:202
#16 0xffffffc010081c24 in ipipe_handle_domain_irq (regs=<optimized out>, hwirq=<optimized out>, domain=<optimized out>)
at ./arch/arm64/include/asm/ipipe.h:160
#17 gic_handle_irq (regs=0xffffffc010cfd400 <ipipe_root>) at drivers/irqchip/irq-gic.c:404
#18 0xffffffc010085e04 in handle_arch_irq_pipelined (regs=<optimized out>) at arch/arm64/kernel/irq.c:43
#19 0xffffffc0100834c8 in el1_irq () at arch/arm64/kernel/entry.S:696
Backtrace stopped: previous frame identical to this frame (corrupt stack?)
(gdb)
--- Core 2 ---
(gdb) info threads
Id Target Id Frame
* 1 Remote target 0x0000007f8722a308 in ?? ()
(gdb) bt
#0 0x0000007f8722a308 in ?? ()
#1 0x000000555c30a678 in ?? ()
#2 0x97cd1a4acef21b00 in ?? ()
#3 0x0000000000000000 in ?? ()
(gdb)
--- Core 3 ---
(gdb) info threads
Id Target Id Frame
* 1 Remote target cpu_relax () at ./arch/arm64/include/asm/processor.h:247
(gdb) bt
#0 cpu_relax () at ./arch/arm64/include/asm/processor.h:247
#1 __read_seqcount_begin (s=<optimized out>) at ./include/linux/seqlock.h:115
#2 raw_read_seqcount_begin (s=<optimized out>) at ./include/linux/seqlock.h:148
#3 read_seqcount_begin (s=<optimized out>) at ./include/linux/seqlock.h:165
#4 ktime_get () at kernel/time/timekeeping.c:757
#5 0xffffffc01014e294 in tick_nohz_irq_enter () at kernel/time/tick-sched.c:1257
#6 tick_irq_enter () at kernel/time/tick-sched.c:1278
#7 0xffffffc0100d3350 in irq_enter () at kernel/softirq.c:354
#8 0xffffffc010185694 in __ipipe_do_sync_stage () at kernel/ipipe/core.c:1696
#9 0xffffffc0101857d0 in __ipipe_sync_stage () at ./include/linux/ipipe.h:116
#10 __ipipe_do_sync_pipeline (top=0xffffffc010cfd400 <ipipe_root>) at kernel/ipipe/core.c:977
#11 0xffffffc010185aec in dispatch_irq_head (irq=<optimized out>) at kernel/ipipe/core.c:1454
#12 0xffffffc010185c10 in __ipipe_dispatch_irq (irq=1, flags=<optimized out>) at kernel/ipipe/core.c:1555
#13 0xffffffc0100814bc in __ipipe_grab_irq (irq=<optimized out>, regs=0x100) at arch/arm64/kernel/ipipe.c:202
#14 0xffffffc010081c24 in ipipe_handle_domain_irq (regs=<optimized out>, hwirq=<optimized out>, domain=<optimized out>)
at ./arch/arm64/include/asm/ipipe.h:160
#15 gic_handle_irq (regs=0x6d2c59) at drivers/irqchip/irq-gic.c:404
#16 0xffffffc010085e04 in handle_arch_irq_pipelined (regs=<optimized out>) at arch/arm64/kernel/irq.c:43
#17 0xffffffc0100834c8 in el1_irq () at arch/arm64/kernel/entry.S:696
Backtrace stopped: previous frame identical to this frame (corrupt stack?)
(gdb)
Best regards,
Josh
next reply other threads:[~2023-12-12 15:07 UTC|newest]
Thread overview: 2+ messages / expand[flat|nested] mbox.gz Atom feed top
2023-12-12 15:07 josh.nestor [this message]
2023-12-15 6:56 ` Resolving a livelock between Xenomai 3 application and Linux kernel using GDB Jan Kiszka
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to='2eRq8h8p8SWhJ6EnmaVwi0DCGy_pWlWDoNQ5lE_y6bQ09xcwtVzNG5Cg7xc24KX92Dj1nULBDiNT9Gj37p3dHriAYApcqWkmvMVUwP8KjIg=@protonmail.com' \
--to=josh.nestor@protonmail.com \
--cc=xenomai@lists.linux.dev \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).