From: Vimal Kumar <vimal.kumar32@gmail.com>
To: "Rafael J. Wysocki" <rafael@kernel.org>
Cc: kernel test robot <oliver.sang@intel.com>,
oe-lkp@lists.linux.dev, lkp@intel.com,
Chinmoy Ghosh <chinmoyghosh2001@gmail.com>,
Mintu Patel <mintupatel89@gmail.com>,
Vishal Badole <badolevishal1116@gmail.com>,
linux-pm@vger.kernel.org, rui.zhang@intel.com,
yu.c.chen@intel.com, Len Brown <len.brown@intel.com>,
Pavel Machek <pavel@ucw.cz>,
Greg Kroah-Hartman <gregkh@linuxfoundation.org>,
linux-kernel@vger.kernel.org
Subject: Re: [PATCH v3] PM / sleep: Mechanism to find source aborting kernel suspend transition
Date: Mon, 5 Feb 2024 10:24:39 +0530 [thread overview]
Message-ID: <20240205045439.GA2553@DESKTOP-KA7F9LU.localdomain> (raw)
In-Reply-To: <CAJZ5v0geH_xn9RDGRUZ1izFt2cOA7_mo8r=8C3HoqZ1icfVKBA@mail.gmail.com>
On Thu, Feb 01, 2024 at 02:59:00PM +0100, Rafael J. Wysocki wrote:
> On Thu, Feb 1, 2024 at 5:44 AM Vimal Kumar <vimal.kumar32@gmail.com> wrote:
> >
> > On Mon, Jan 29, 2024 at 10:08:32PM +0800, kernel test robot wrote:
> > >
> > >
> > > Hello,
> > >
B> > > kernel test robot noticed "BUG:sleeping_function_called_from_invalid_context_at_include/linux/sched/mm.h" on:
> > >
> > > commit: e2d228456d500f3b00ad746b353fa85eee235b7a ("[PATCH v3] PM / sleep: Mechanism to find source aborting kernel suspend transition")
> > > url: https://github.com/intel-lab-lkp/linux/commits/Vimal-Kumar/PM-sleep-Mechanism-to-find-source-aborting-kernel-suspend-transition/20240109-210519
> > > base: https://git.kernel.org/cgit/linux/kernel/git/rafael/linux-pm.git linux-next
> > > patch link: https://lore.kernel.org/all/20240109130210.31938-1-vimal.kumar32@gmail.com/
> > > patch subject: [PATCH v3] PM / sleep: Mechanism to find source aborting kernel suspend transition
> > >
> > > in testcase: suspend-stress
> > > version:
> > > with following parameters:
> > >
> > > mode: freeze
> > > iterations: 10
> > >
> > >
> > >
> > > compiler: gcc-12
> > > test machine: 4 threads (Broadwell) with 8G memory
> > >
> > > (please refer to attached dmesg/kmsg for entire log/backtrace)
> > >
> > >
> > >
> > > If you fix the issue in a separate patch/commit (i.e. not just a new version of
> > > the same patch/commit), kindly add following tags
> > > | Reported-by: kernel test robot <oliver.sang@intel.com>
> > > | Closes: https://lore.kernel.org/oe-lkp/202401292109.f115a688-oliver.sang@intel.com
> > >
> > >
> > > kern :err : [ 97.935754] BUG: sleeping function called from invalid context at include/linux/sched/mm.h:306
> > > kern :err : [ 97.935886] in_atomic(): 1, irqs_disabled(): 1, non_block: 0, pid: 0, name: swapper/1
> > > kern :err : [ 97.936000] preempt_count: 10002, expected: 0
> > > kern :err : [ 97.936071] RCU nest depth: 0, expected: 0
> > > kern :warn : [ 97.936138] CPU: 1 PID: 0 Comm: swapper/1 Not tainted 6.7.0-rc8-00169-ge2d228456d50 #1
> > > kern :warn : [ 97.936255] Hardware name: /NUC5i3RYB, BIOS RYBDWi35.86A.0363.2017.0316.1028 03/16/2017
> > > kern :warn : [ 97.936371] Call Trace:
> > > kern :warn : [ 97.936419] <IRQ>
> > > kern :warn : [ 97.936461] dump_stack_lvl (lib/dump_stack.c:107 (discriminator 1))
> > > kern :warn : [ 97.936529] __might_resched (kernel/sched/core.c:10152)
> > > kern :warn : [ 97.936598] ? preempt_notifier_dec (kernel/sched/core.c:10106)
> > > kern :warn : [ 97.936672] __kmem_cache_alloc_node (include/linux/kernel.h:107 include/linux/sched/mm.h:306 mm/slab.h:710 mm/slub.c:3460 mm/slub.c:3517)
> > > kern :warn : [ 97.936750] ? kasprintf (lib/kasprintf.c:54)
> > > kern :warn : [ 97.936812] ? pointer (lib/vsprintf.c:2755)
> > > kern :warn : [ 97.936875] ? kasprintf (lib/kasprintf.c:54)
> > > kern :warn : [ 97.936936] __kmalloc_node_track_caller (include/linux/kasan.h:198 mm/slab_common.c:1007 mm/slab_common.c:1027)
> > > kern :warn : [ 97.937017] kvasprintf (lib/kasprintf.c:25)
> > > kern :warn : [ 97.937079] ? bust_spinlocks (lib/kasprintf.c:16)
> > > kern :warn : [ 97.937146] ? enqueue_hrtimer (kernel/time/hrtimer.c:1095 (discriminator 3))
> > > kern :warn : [ 97.937214] ? tick_nohz_highres_handler (kernel/time/tick-sched.c:1530)
> > > kern :warn : [ 97.937295] kasprintf (lib/kasprintf.c:54)
> > > kern :warn : [ 97.937354] ? kvasprintf_const (lib/kasprintf.c:54)
> > > kern :warn : [ 97.937426] ? _raw_spin_lock_irqsave (arch/x86/include/asm/atomic.h:115 include/linux/atomic/atomic-arch-fallback.h:2164 include/linux/atomic/atomic-instrumented.h:1296 include/asm-generic/qspinlock.h:111 include/linux/spinlock.h:187 include/linux/spinlock_api_smp.h:111 kernel/locking/spinlock.c:162)
> > > kern :warn : [ 97.937501] ? _raw_read_unlock_irqrestore (kernel/locking/spinlock.c:161)
> > > kern :warn : [ 97.937581] ? _raw_spin_lock (arch/x86/include/asm/atomic.h:115 include/linux/atomic/atomic-arch-fallback.h:2164 include/linux/atomic/atomic-instrumented.h:1296 include/asm-generic/qspinlock.h:111 include/linux/spinlock.h:187 include/linux/spinlock_api_smp.h:134 kernel/locking/spinlock.c:154)
> > > kern :warn : [ 97.937648] ? _raw_write_lock_irq (kernel/locking/spinlock.c:153)
> > > kern :warn : [ 97.937719] ? irq_pm_check_wakeup (kernel/irq/pm.c:24)
> > > kern :warn : [ 97.937793] pm_system_wakeup (drivers/base/power/wakeup.c:981)
> > > kern :warn : [ 97.937862] irq_pm_check_wakeup (kernel/irq/pm.c:24)
> > > kern :warn : [ 97.937933] handle_fasteoi_irq (kernel/irq/chip.c:518 kernel/irq/chip.c:502 kernel/irq/chip.c:698)
> > > kern :warn : [ 97.938006] __common_interrupt (arch/x86/kernel/irq.c:271 (discriminator 22))
> > > kern :warn : [ 97.938078] common_interrupt (arch/x86/kernel/irq.c:247 (discriminator 14))
> > > kern :warn : [ 97.938145] </IRQ>
> > > kern :warn : [ 97.938187] <TASK>
> > > kern :warn : [ 97.938230] asm_common_interrupt (arch/x86/include/asm/idtentry.h:640)
> > > kern :warn : [ 97.938301] RIP: 0010:cpuidle_enter_s2idle (arch/x86/include/asm/irqflags.h:78 drivers/cpuidle/cpuidle.c:199)
> > > kern :warn : [ 97.938384] Code: d9 75 83 85 d2 7f 11 48 83 c4 20 89 d0 5b 5d 41 5c 41 5d 41 5e 41 5f c3 4c 89 c6 48 89 cf 89 14 24 e8 fb 64 9b 00 fb 8b 14 24 <eb> db 31 d2 eb d7 48 89 0c 24 48 89 74 24 08 e8 03 51 9f fe 4c 8b
> > > All code
> > > ========
> > > 0: d9 75 83 fnstenv -0x7d(%rbp)
> > > 3: 85 d2 test %edx,%edx
> > > 5: 7f 11 jg 0x18
> > > 7: 48 83 c4 20 add $0x20,%rsp
> > > b: 89 d0 mov %edx,%eax
> > > d: 5b pop %rbx
> > > e: 5d pop %rbp
> > > f: 41 5c pop %r12
> > > 11: 41 5d pop %r13
> > > 13: 41 5e pop %r14
> > > 15: 41 5f pop %r15
> > > 17: c3 retq
> > > 18: 4c 89 c6 mov %r8,%rsi
> > > 1b: 48 89 cf mov %rcx,%rdi
> > > 1e: 89 14 24 mov %edx,(%rsp)
> > > 21: e8 fb 64 9b 00 callq 0x9b6521
> > > 26: fb sti
> > > 27: 8b 14 24 mov (%rsp),%edx
> > > 2a:* eb db jmp 0x7 <-- trapping instruction
> > > 2c: 31 d2 xor %edx,%edx
> > > 2e: eb d7 jmp 0x7
> > > 30: 48 89 0c 24 mov %rcx,(%rsp)
> > > 34: 48 89 74 24 08 mov %rsi,0x8(%rsp)
> > > 39: e8 03 51 9f fe callq 0xfffffffffe9f5141
> > > 3e: 4c rex.WR
> > > 3f: 8b .byte 0x8b
> > >
> > > Code starting with the faulting instruction
> > > ===========================================
> > > 0: eb db jmp 0xffffffffffffffdd
> > > 2: 31 d2 xor %edx,%edx
> > > 4: eb d7 jmp 0xffffffffffffffdd
> > > 6: 48 89 0c 24 mov %rcx,(%rsp)
> > > a: 48 89 74 24 08 mov %rsi,0x8(%rsp)
> > > f: e8 03 51 9f fe callq 0xfffffffffe9f5117
> > > 14: 4c rex.WR
> > > 15: 8b .byte 0x8b
> > > kern :warn : [ 97.938632] RSP: 0018:ffffc90000177db8 EFLAGS: 00000286
> > > kern :warn : [ 97.938717] RAX: 00000000037a402c RBX: 0000000000000009 RCX: 0000000000000000
> > > kern :warn : [ 97.938823] RDX: 0000000000000008 RSI: 0000000937e23ad2 RDI: 0000000000000000
> > > kern :warn : [ 97.938927] RBP: ffffffff84f7f680 R08: 0000000000000000 R09: ffffed1021ad5530
> > > kern :warn : [ 97.939032] R10: ffff88810d6aa987 R11: 0000000000000000 R12: ffff88820decca10
> > > kern :warn : [ 97.939137] R13: dffffc0000000000 R14: 000000000027ac40 R15: 000000000027ac40
> > > kern :warn : [ 97.939244] ? cpuidle_enter_s2idle (arch/x86/include/asm/irqflags.h:42 arch/x86/include/asm/irqflags.h:77 drivers/cpuidle/cpuidle.c:199)
> > > kern :warn : [ 97.939319] cpuidle_idle_call (kernel/sched/idle.c:190)
> > > kern :warn : [ 97.939389] ? arch_cpu_idle_exit+0x40/0x40
> > > kern :warn : [ 97.939459] ? mark_tsc_async_resets (arch/x86/kernel/tsc_sync.c:51)
> > > kern :warn : [ 97.939536] do_idle (kernel/sched/idle.c:282)
> > > kern :warn : [ 97.939594] cpu_startup_entry (kernel/sched/idle.c:379 (discriminator 1))
> > > kern :warn : [ 97.939662] start_secondary (arch/x86/kernel/smpboot.c:224 arch/x86/kernel/smpboot.c:304)
> > > kern :warn : [ 97.939731] ? set_cpu_sibling_map (arch/x86/kernel/smpboot.c:254)
> > > kern :warn : [ 97.939808] ? soft_restart_cpu (arch/x86/kernel/head_64.S:485)
> > > kern :warn : [ 97.939878] secondary_startup_64_no_verify (arch/x86/kernel/head_64.S:448)
> > > kern :warn : [ 97.939963] </TASK>
> > > kern :info : [ 97.948803] bdw_uncore 0000:00:00.0: PM: calling pci_pm_resume_noirq+0x0/0x390 @ 85, parent: 00
> > > kern :info : [ 97.948993] bdw_uncore 0000:00:00.0: PM: pci_pm_resume_noirq+0x0/0x390 returned 0 after 17 usecs
> > >
> > >
> > >
> > > The kernel config and materials to reproduce are available at:
> > > https://download.01.org/0day-ci/archive/20240129/202401292109.f115a688-oliver.sang@intel.com
> > >
> > >
> > >
> > > --
> > > 0-DAY CI Kernel Test Service
> > > https://github.com/intel/lkp-tests/wiki
> > >
> >
> > Hi Greg,
> >
> > This is an example where "pm_system_wakeup" can be called from atomic context. It also gives an actual instance where we can utilise this feature to find the source.
> >
> > The bug reported due to "GFP_KERNEL" flag, but it will be applicable for using "mutex_lock" as well.
> > I will change to ATOMIC flag in kmalloc, as well as use raw_spin_lock* instead of mutex_lock.
>
> Maybe instead of trying to add bandaids, you could work on making the
> wakeup sources infrastructure suitable for addressing your use case?
Sure, Rafael. Thanks for acknowledging it. We are making a plan to correct wakeup sources infrastructure but this patch will help those products
which are already released and seeking for "how to debug and fix".
As per my understanding, this current patch will help find the issues in projects that are based on <6.8 mainline, and
the developer/OEM/ODM can find and fix the issues easily.
Warm Regards,
Vimal Kumar
prev parent reply other threads:[~2024-02-05 4:54 UTC|newest]
Thread overview: 4+ messages / expand[flat|nested] mbox.gz Atom feed top
[not found] <20240109130210.31938-1-vimal.kumar32@gmail.com>
2024-01-29 14:08 ` [PATCH v3] PM / sleep: Mechanism to find source aborting kernel suspend transition kernel test robot
2024-02-01 4:44 ` Vimal Kumar
2024-02-01 13:59 ` Rafael J. Wysocki
2024-02-05 4:54 ` Vimal Kumar [this message]
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20240205045439.GA2553@DESKTOP-KA7F9LU.localdomain \
--to=vimal.kumar32@gmail.com \
--cc=badolevishal1116@gmail.com \
--cc=chinmoyghosh2001@gmail.com \
--cc=gregkh@linuxfoundation.org \
--cc=len.brown@intel.com \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-pm@vger.kernel.org \
--cc=lkp@intel.com \
--cc=mintupatel89@gmail.com \
--cc=oe-lkp@lists.linux.dev \
--cc=oliver.sang@intel.com \
--cc=pavel@ucw.cz \
--cc=rafael@kernel.org \
--cc=rui.zhang@intel.com \
--cc=yu.c.chen@intel.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).