* Re: [PATCH v6 20/20] irqdomain: Switch to per-domain locking
[not found] ` <20230213104302.17307-21-johan+linaro@kernel.org>
@ 2023-03-07 13:51 ` David Woodhouse
2023-03-07 14:06 ` Juergen Gross
0 siblings, 1 reply; 3+ messages in thread
From: David Woodhouse @ 2023-03-07 13:51 UTC (permalink / raw
To: Johan Hovold, Marc Zyngier, Thomas Gleixner, xen-devel
Cc: x86, platform-driver-x86, linux-arm-kernel, linux-mips,
linux-kernel, Hsin-Yi Wang, Mark-PK Tsai
[-- Attachment #1: Type: text/plain, Size: 4948 bytes --]
On Mon, 2023-02-13 at 11:43 +0100, Johan Hovold wrote:
> The IRQ domain structures are currently protected by the global
> irq_domain_mutex. Switch to using more fine-grained per-domain locking,
> which can speed up parallel probing by reducing lock contention.
>
> On a recent arm64 laptop, the total time spent waiting for the locks
> during boot drops from 160 to 40 ms on average, while the maximum
> aggregate wait time drops from 550 to 90 ms over ten runs for example.
>
> Note that the domain lock of the root domain (innermost domain) must be
> used for hierarchical domains. For non-hierarchical domains (as for root
> domains), the new root pointer is set to the domain itself so that
> &domain->root->mutex always points to the right lock.
>
> Also note that hierarchical domains should be constructed using
> irq_domain_create_hierarchy() (or irq_domain_add_hierarchy()) to avoid
> having racing allocations access a not fully initialised domain. As a
> safeguard, the lockdep assertion in irq_domain_set_mapping() will catch
> any offenders that also fail to set the root domain pointer.
>
> Tested-by: Hsin-Yi Wang <hsinyi@chromium.org>
> Tested-by: Mark-PK Tsai <mark-pk.tsai@mediatek.com>
> Signed-off-by: Johan Hovold <johan+linaro@kernel.org>
Broke Xen. And it's *so* easy to test. As long as you have qemu master
branch from no older than last Thursday, that is...
$ qemu-system-x86_64 -serial mon:stdio -display none \
-accel kvm,xen-version=0x4000e,kernel-irqchip=split \
-kernel arch/x86/boot/bzImage -append "console=ttyS0"
...
[ 0.466554] BUG: kernel NULL pointer dereference, address: 00000000000000c0
[ 0.467249] #PF: supervisor read access in kernel mode
[ 0.467249] #PF: error_code(0x0000) - not-present page
[ 0.467249] PGD 0 P4D 0
[ 0.467249] Oops: 0000 [#1] PREEMPT SMP PTI
[ 0.467249] CPU: 0 PID: 1 Comm: swapper/0 Not tainted 6.2.0-rc4+ #1206
[ 0.467249] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.16.1-0-g3208b098f51a-prebuilt.qemu.org 04/01/2014
[ 0.467249] RIP: 0010:irq_domain_create_hierarchy+0x2c/0x70
[ 0.467249] Code: 1e fa 0f 1f 44 00 00 41 54 49 89 fc 48 89 cf 55 89 f5 53 85 d2 74 40 89 d6 31 c9 89 d2 e8 2c fa ff ff 48 89 c3 48 85 db 74 21 <49> 8b 84 24 c0 00 00 00 09 6b 28 48 89 df 4c 89 a3 f0 00 00 00 48
[ 0.467249] RSP: 0000:ffffc90000013e60 EFLAGS: 00010286
[ 0.467249] RAX: ffff8880053a1a00 RBX: ffff8880053a1a00 RCX: 0000000000000000
[ 0.467249] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffffffff84828fa0
[ 0.467249] RBP: 0000000000000010 R08: 0000000000000003 R09: 0000000000000000
[ 0.467249] R10: 0000000025a89be7 R11: 00000000442a63fa R12: 0000000000000000
[ 0.467249] R13: ffffffff83ac1b98 R14: 0000000000000000 R15: 0000000000000000
[ 0.467249] FS: 0000000000000000(0000) GS:ffff888007a00000(0000) knlGS:0000000000000000
[ 0.467249] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 0.467249] CR2: 00000000000000c0 CR3: 0000000002824000 CR4: 00000000000006f0
[ 0.467249] Call Trace:
[ 0.467249] <TASK>
[ 0.467249] ? __pfx_pci_arch_init+0x10/0x10
[ 0.467249] __msi_create_irq_domain+0x85/0x170
[ 0.467249] ? __pfx_pci_arch_init+0x10/0x10
[ 0.467249] xen_create_pci_msi_domain+0x34/0x40
[ 0.467249] x86_create_pci_msi_domain+0x12/0x1e
[ 0.467249] pci_arch_init+0x31/0x7a
[ 0.467249] ? __pfx_pci_arch_init+0x10/0x10
[ 0.467249] do_one_initcall+0x5f/0x320
[ 0.467249] ? rcu_read_lock_sched_held+0x43/0x80
[ 0.467249] kernel_init_freeable+0x189/0x1c6
[ 0.467249] ? __pfx_kernel_init+0x10/0x10
[ 0.467249] kernel_init+0x1a/0x130
[ 0.467249] ret_from_fork+0x2c/0x50
[ 0.467249] </TASK>
[ 0.467249] Modules linked in:
[ 0.467249] CR2: 00000000000000c0
[ 0.467249] ---[ end trace 0000000000000000 ]---
[ 0.467249] RIP: 0010:irq_domain_create_hierarchy+0x2c/0x70
[ 0.467249] Code: 1e fa 0f 1f 44 00 00 41 54 49 89 fc 48 89 cf 55 89 f5 53 85 d2 74 40 89 d6 31 c9 89 d2 e8 2c fa ff ff 48 89 c3 48 85 db 74 21 <49> 8b 84 24 c0 00 00 00 09 6b 28 48 89 df 4c 89 a3 f0 00 00 00 48
[ 0.467249] RSP: 0000:ffffc90000013e60 EFLAGS: 00010286
[ 0.467249] RAX: ffff8880053a1a00 RBX: ffff8880053a1a00 RCX: 0000000000000000
[ 0.467249] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffffffff84828fa0
[ 0.467249] RBP: 0000000000000010 R08: 0000000000000003 R09: 0000000000000000
[ 0.467249] R10: 0000000025a89be7 R11: 00000000442a63fa R12: 0000000000000000
[ 0.467249] R13: ffffffff83ac1b98 R14: 0000000000000000 R15: 0000000000000000
[ 0.467249] FS: 0000000000000000(0000) GS:ffff888007a00000(0000) knlGS:0000000000000000
[ 0.467249] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 0.467249] CR2: 00000000000000c0 CR3: 0000000002824000 CR4: 00000000000006f0
[ 0.467249] Kernel panic - not syncing: Fatal exception
[-- Attachment #2: smime.p7s --]
[-- Type: application/pkcs7-signature, Size: 5965 bytes --]
^ permalink raw reply [flat|nested] 3+ messages in thread
* Re: [PATCH v6 20/20] irqdomain: Switch to per-domain locking
2023-03-07 13:51 ` [PATCH v6 20/20] irqdomain: Switch to per-domain locking David Woodhouse
@ 2023-03-07 14:06 ` Juergen Gross
2023-03-07 14:18 ` David Woodhouse
0 siblings, 1 reply; 3+ messages in thread
From: Juergen Gross @ 2023-03-07 14:06 UTC (permalink / raw
To: David Woodhouse, Johan Hovold, Marc Zyngier, Thomas Gleixner,
xen-devel
Cc: x86, platform-driver-x86, linux-arm-kernel, linux-mips,
linux-kernel, Hsin-Yi Wang, Mark-PK Tsai
[-- Attachment #1.1.1: Type: text/plain, Size: 1418 bytes --]
On 07.03.23 14:51, David Woodhouse wrote:
> On Mon, 2023-02-13 at 11:43 +0100, Johan Hovold wrote:
>> The IRQ domain structures are currently protected by the global
>> irq_domain_mutex. Switch to using more fine-grained per-domain locking,
>> which can speed up parallel probing by reducing lock contention.
>>
>> On a recent arm64 laptop, the total time spent waiting for the locks
>> during boot drops from 160 to 40 ms on average, while the maximum
>> aggregate wait time drops from 550 to 90 ms over ten runs for example.
>>
>> Note that the domain lock of the root domain (innermost domain) must be
>> used for hierarchical domains. For non-hierarchical domains (as for root
>> domains), the new root pointer is set to the domain itself so that
>> &domain->root->mutex always points to the right lock.
>>
>> Also note that hierarchical domains should be constructed using
>> irq_domain_create_hierarchy() (or irq_domain_add_hierarchy()) to avoid
>> having racing allocations access a not fully initialised domain. As a
>> safeguard, the lockdep assertion in irq_domain_set_mapping() will catch
>> any offenders that also fail to set the root domain pointer.
>>
>> Tested-by: Hsin-Yi Wang <hsinyi@chromium.org>
>> Tested-by: Mark-PK Tsai <mark-pk.tsai@mediatek.com>
>> Signed-off-by: Johan Hovold <johan+linaro@kernel.org>
>
> Broke Xen.
Fixed with commit ad32ab9604f2.
Juergen
[-- Attachment #1.1.2: OpenPGP public key --]
[-- Type: application/pgp-keys, Size: 3149 bytes --]
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 495 bytes --]
^ permalink raw reply [flat|nested] 3+ messages in thread
* Re: [PATCH v6 20/20] irqdomain: Switch to per-domain locking
2023-03-07 14:06 ` Juergen Gross
@ 2023-03-07 14:18 ` David Woodhouse
0 siblings, 0 replies; 3+ messages in thread
From: David Woodhouse @ 2023-03-07 14:18 UTC (permalink / raw
To: Juergen Gross, Johan Hovold, Marc Zyngier, Thomas Gleixner,
xen-devel
Cc: x86, platform-driver-x86, linux-arm-kernel, linux-mips,
linux-kernel, Hsin-Yi Wang, Mark-PK Tsai
[-- Attachment #1: Type: text/plain, Size: 1555 bytes --]
On Tue, 2023-03-07 at 15:06 +0100, Juergen Gross wrote:
> On 07.03.23 14:51, David Woodhouse wrote:
> > On Mon, 2023-02-13 at 11:43 +0100, Johan Hovold wrote:
> > > The IRQ domain structures are currently protected by the global
> > > irq_domain_mutex. Switch to using more fine-grained per-domain locking,
> > > which can speed up parallel probing by reducing lock contention.
> > >
> > > On a recent arm64 laptop, the total time spent waiting for the locks
> > > during boot drops from 160 to 40 ms on average, while the maximum
> > > aggregate wait time drops from 550 to 90 ms over ten runs for example.
> > >
> > > Note that the domain lock of the root domain (innermost domain) must be
> > > used for hierarchical domains. For non-hierarchical domains (as for root
> > > domains), the new root pointer is set to the domain itself so that
> > > &domain->root->mutex always points to the right lock.
> > >
> > > Also note that hierarchical domains should be constructed using
> > > irq_domain_create_hierarchy() (or irq_domain_add_hierarchy()) to avoid
> > > having racing allocations access a not fully initialised domain. As a
> > > safeguard, the lockdep assertion in irq_domain_set_mapping() will catch
> > > any offenders that also fail to set the root domain pointer.
> > >
> > > Tested-by: Hsin-Yi Wang <hsinyi@chromium.org>
> > > Tested-by: Mark-PK Tsai <mark-pk.tsai@mediatek.com>
> > > Signed-off-by: Johan Hovold <johan+linaro@kernel.org>
> >
> > Broke Xen.
>
> Fixed with commit ad32ab9604f2.
Thanks.
[-- Attachment #2: smime.p7s --]
[-- Type: application/pkcs7-signature, Size: 5965 bytes --]
^ permalink raw reply [flat|nested] 3+ messages in thread
end of thread, other threads:[~2023-03-07 14:18 UTC | newest]
Thread overview: 3+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
[not found] <20230213104302.17307-1-johan+linaro@kernel.org>
[not found] ` <20230213104302.17307-21-johan+linaro@kernel.org>
2023-03-07 13:51 ` [PATCH v6 20/20] irqdomain: Switch to per-domain locking David Woodhouse
2023-03-07 14:06 ` Juergen Gross
2023-03-07 14:18 ` David Woodhouse
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).