* bugzilla #197 fast create/destroy BUG_ON()
@ 2005-09-09 22:07 Ryan Harper
2005-09-12 21:27 ` [PATCH] workaround for bug#197 Ryan Harper
0 siblings, 1 reply; 6+ messages in thread
From: Ryan Harper @ 2005-09-09 22:07 UTC (permalink / raw
To: xen-devel
I haven't made any further progress on this [1]bug, even with adding in
some extra tracing.
When we put xm create following by xm destroy in a loop, eventually xen
hits a hard reboot dumping out a BUG_ON()
(XEN) BUG at domain.c:1054
(XEN) CPU: 0
(XEN) EIP: e008:[<ff12a6b6>] domain_relinquish_resources+0x43/0x1c8
(XEN) EFLAGS: 00010282 CONTEXT: hypervisor
(XEN) eax: ff187fb8 ebx: ffbf1080 ecx: ffbf4000 edx: 00000000
(XEN) esi: 00000007 edi: ff103fac ebp: ff103b0c esp: ff103af4
(XEN) cr0: 8005003b cr3: db629000
(XEN) ds: e010 es: e010 fs: 0000 gs: 0000 ss: e010 cs: e008
(XEN) Xen stack trace from esp=ff103af4:
(XEN) ff17ba49 ff17bb67 0000041e ff11cfa9 ffbd9094 00000000 ff103b2c ff109027
(XEN) ffbda080 ffbda308 00000001 ffbda080 ffbda080 00000000 ff103f8c ff107f88
(XEN) ffbda080 ff103dbc 00000000 ffbda310 00000001 00000001 ff103b5c ff13f087
(XEN) 00000003 00000000 ffbc7080 ffbda080 ff103b74 00000020 ff103b8c ff10c368
(XEN) 00000000 ffbf10d4 ff103b9c ff130211 00000001 00000001 00000001 00000004
(XEN) ffbf1e00 ff103cc8 ff111a6c ff103c2c 00000001 00000009 ff103bbc 00000004
(XEN) 00000004 ff103fb4 ff103bcc ff1233a3 00000004 80000003 80000003 00000004
(XEN) 80000002 80000003 fd8e671c ff12fb9a ff103be8 80000004 ffbf2080 ffbf2080
(XEN) fd750ef4 ff103fac ff103c5c 80000003 00ef0000 80000003 80000003 80000003
(XEN) 80000002 ff103c48 ff1352eb fd8e6710 ff103c34 00000020 ff103c4c 00000001
(XEN) 80000003 00000001 ffbf2080 ffbf2080 ffbf2080 80000003 80000004 80000003
(XEN) ffbf2080 00000000 ff103c8c ff1350ed fd70c770 ffbf2080 ffbf2080 00000540
(XEN) c9e72063 00000000 00000001 00000001 00000000 000001d1 ff103cbc ff135407
(XEN) fedd1000 ffbf2080 00000400 00000001 00000004 ffbf1e00 ff103ddc ff111a6c
(XEN) ff103d40 00000001 00000008 ff135de0 fd6edab0 00000001 ff103cdc ff13b552
(XEN) 20000000 00000000 0000b400 000004c4 c4b40000 00000004 00008000 00009b42
(XEN) 000007dc 00000000 f0000000 04c4b400 00000004 00000001 33ef003c 33ef003c
(XEN) 00000000 00000000 00000000 00000000 000007dc 8000003e 8000003f 8000003e
(XEN) ffbf2080 00000000 00000000 00009b42 00008000 33ef003c 00000000 33ef003c
(XEN) 00000000 00000000 00000000 0000755a 00000000 23ef0000 ff103d7c ff12315c
(XEN) Xen call trace:
(XEN) [<ff12a6b6>] domain_relinquish_resources+0x43/0x1c8
(XEN) [<ff109027>] domain_kill+0x62/0x9e
(XEN) [<ff107f88>] do_dom0_op+0x54d/0x103b
(XEN) [<ff155d8f>] hypercall+0x8f/0xaf
the line in question is:
BUG_ON(!cpus_empty(d->cpumask));
This says to me, bail if the domain's cpumask is NOT empty. AFAICT,
the only places that cpumask for a domain are modified are:
1. startup_cpu_idle_loop() in xen/arch/x86/domain.c
cpu_set(smp_processor_id(),v->domain->cpumask);
2. __context_switch() in xen/arch/x86/domain.c
if ( p->domain != n->domain )
cpu_set(cpu, n->domain->cpumask);
...
if ( p->domain != n->domain )
cpu_clear(cpu, p->domain->cpumask);
If we are hitting that assert, then one or more of the vcpus in the
domain are still running? Any help on pointing out where I've
misunderstood whats happening or good places to insert some debugging
would be of great help.
1. http://bugzilla.xensource.com/bugzilla/show_bug.cgi?id=197
--
Ryan Harper
Software Engineer; Linux Technology Center
IBM Corp., Austin, Tx
(512) 838-9253 T/L: 678-9253
ryanh@us.ibm.com
^ permalink raw reply [flat|nested] 6+ messages in thread
* [PATCH] workaround for bug#197
2005-09-09 22:07 bugzilla #197 fast create/destroy BUG_ON() Ryan Harper
@ 2005-09-12 21:27 ` Ryan Harper
2005-09-12 23:07 ` Dan Smith
2005-09-13 18:22 ` [PATCH] workaround for bug#197: second try Ryan Harper
0 siblings, 2 replies; 6+ messages in thread
From: Ryan Harper @ 2005-09-12 21:27 UTC (permalink / raw
To: xen-devel
I don't think this is the right fix, but it does highlight the issue.
While killing a domain, the vcpus are descheduled, but every
now and then, one of the cpus is still running one of the vcpus, which
means d->cpumask is not empty. This triggers the BUG_ON() in
xen/arch/x86/domain.c:domain_relinquish_resources(). The patch puts in
some printks and a cpu_relax() loop till the cpumask is empty before
calling domain_relinquish_resources(). With this patch, I've gone
through several thousand iterations of create/destroy without crashing.
--
Ryan Harper
Software Engineer; Linux Technology Center
IBM Corp., Austin, Tx
(512) 838-9253 T/L: 678-9253
ryanh@us.ibm.com
diffstat output:
domain.c | 8 ++++++++
1 files changed, 8 insertions(+)
Signed-off-by: Ryan Harper <ryanh@us.ibm.com>
---
diff -r 413c911e5780 xen/common/domain.c
--- a/xen/common/domain.c Mon Sep 12 12:48:33 2005
+++ b/xen/common/domain.c Mon Sep 12 13:25:07 2005
@@ -112,6 +112,14 @@
{
for_each_vcpu(d, v)
sched_rem_domain(v);
+ if(!cpus_empty(d->cpumask)) {
+ printk("ACK! DOM%d still running, waiting before dying\n",
+ d->domain_id);
+ while(!cpus_empty(d->cpumask))
+ cpu_relax();
+ printk("DOM%d cpumask clear, relinquishing resources\n",
+ d->domain_id);
+ }
domain_relinquish_resources(d);
put_domain(d);
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [PATCH] workaround for bug#197
2005-09-12 21:27 ` [PATCH] workaround for bug#197 Ryan Harper
@ 2005-09-12 23:07 ` Dan Smith
2005-09-13 18:22 ` [PATCH] workaround for bug#197: second try Ryan Harper
1 sibling, 0 replies; 6+ messages in thread
From: Dan Smith @ 2005-09-12 23:07 UTC (permalink / raw
To: Ryan Harper; +Cc: xen-devel
RH> With this patch, I've gone through several thousand iterations of
RH> create/destroy without crashing.
The test 10_create_fastdestroy in the xm-test suite pokes this bug
every time for me. After applying this patch, I get the following on
a serial console while running the test:
(XEN) ACK! DOM1 still running, waiting before dying
After that, the system is completely hung.
--
Dan Smith
IBM Linux Technology Center
Open Hypervisor Team
email: danms@us.ibm.com
^ permalink raw reply [flat|nested] 6+ messages in thread
* [PATCH] workaround for bug#197: second try
2005-09-12 21:27 ` [PATCH] workaround for bug#197 Ryan Harper
2005-09-12 23:07 ` Dan Smith
@ 2005-09-13 18:22 ` Ryan Harper
2005-09-13 19:04 ` Keir Fraser
2005-09-13 19:23 ` Dan Smith
1 sibling, 2 replies; 6+ messages in thread
From: Ryan Harper @ 2005-09-13 18:22 UTC (permalink / raw
To: xen-devel; +Cc: Dan Smith, Puthiyaparambil, Aravindh
Thanks for trying out the last patch. While the previous workaround worked for
me, it did not for others. Looking into domain_pause(), we can see that
vcpu_sleep_sync() is called on each vcpu:
/*
* We can be sure that the VCPU is finally descheduled after the running
* flag is cleared and the scheduler lock is released.
*/
while ( test_bit(_VCPUF_running, &v->vcpu_flags)
&& !domain_runnable(v)
&& spin_is_locked(&schedule_data[v->processor].schedule_lock) )
cpu_relax();
If we are to believe the comment, (which makes sense), then the while loop code
is broken. That is, this function will spin until *any* of the three tests
returns false rather than waiting until *all* tests are false. This patch
switches the &&s to ||s and inverts the domain_runnable() check. I believe we
want to spin while 1) vcpu_running flag is up 2) the domain is runnable and 3)
the scheduler lock is held.
--
Ryan Harper
Software Engineer; Linux Technology Center
IBM Corp., Austin, Tx
(512) 838-9253 T/L: 678-9253
ryanh@us.ibm.com
diffstat output:
schedule.c | 4 ++--
1 files changed, 2 insertions(+), 2 deletions(-)
---
#
# fix logic to match comments. ie. we want to spin until
# 1) the running flag is down,
# 2) the domain isnt runnable (pausecnt > 0)
# 3) the scheduler lock isnt held
#
# Signed-off-by: Ryan Harper <ryanh@us.ibm.com>
#
diff -r 413c911e5780 xen/common/schedule.c
--- a/xen/common/schedule.c Mon Sep 12 12:48:33 2005
+++ b/xen/common/schedule.c Tue Sep 13 09:46:36 2005
@@ -214,8 +214,8 @@
* flag is cleared and the scheduler lock is released.
*/
while ( test_bit(_VCPUF_running, &v->vcpu_flags)
- && !domain_runnable(v)
- && spin_is_locked(&schedule_data[v->processor].schedule_lock) )
+ || domain_runnable(v)
+ || spin_is_locked(&schedule_data[v->processor].schedule_lock) )
cpu_relax();
sync_vcpu_execstate(v);
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [PATCH] workaround for bug#197: second try
2005-09-13 18:22 ` [PATCH] workaround for bug#197: second try Ryan Harper
@ 2005-09-13 19:04 ` Keir Fraser
2005-09-13 19:23 ` Dan Smith
1 sibling, 0 replies; 6+ messages in thread
From: Keir Fraser @ 2005-09-13 19:04 UTC (permalink / raw
To: Ryan Harper; +Cc: Dan Smith, xen-devel, Puthiyaparambil, Aravindh
On 13 Sep 2005, at 19:22, Ryan Harper wrote:
>
> If we are to believe the comment, (which makes sense), then the while
> loop code
> is broken. That is, this function will spin until *any* of the three
> tests
> returns false rather than waiting until *all* tests are false. This
> patch
> switches the &&s to ||s and inverts the domain_runnable() check. I
> believe we
> want to spin while 1) vcpu_running flag is up 2) the domain is
> runnable and 3)
> the scheduler lock is held.
Good catch. Actually we want to spin only while the domain is
non-runnable (just to catch the very unlikely event that someone else
wakes the domain while we spin). I updated the comment and the loop
predicate and checked the patch in. Thanks!
-- Keir
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [PATCH] workaround for bug#197: second try
2005-09-13 18:22 ` [PATCH] workaround for bug#197: second try Ryan Harper
2005-09-13 19:04 ` Keir Fraser
@ 2005-09-13 19:23 ` Dan Smith
1 sibling, 0 replies; 6+ messages in thread
From: Dan Smith @ 2005-09-13 19:23 UTC (permalink / raw
To: Ryan Harper; +Cc: xen-devel, Puthiyaparambil, Aravindh
RH> This patch switches the &&s to ||s and inverts the
RH> domain_runnable() check. I believe we want to spin while 1)
RH> vcpu_running flag is up 2) the domain is runnable and 3) the
RH> scheduler lock is held.
This fixed it for me as well... Thanks Ryan!
--
Dan Smith
IBM Linux Technology Center
Open Hypervisor Team
email: danms@us.ibm.com
^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2005-09-13 19:23 UTC | newest]
Thread overview: 6+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2005-09-09 22:07 bugzilla #197 fast create/destroy BUG_ON() Ryan Harper
2005-09-12 21:27 ` [PATCH] workaround for bug#197 Ryan Harper
2005-09-12 23:07 ` Dan Smith
2005-09-13 18:22 ` [PATCH] workaround for bug#197: second try Ryan Harper
2005-09-13 19:04 ` Keir Fraser
2005-09-13 19:23 ` Dan Smith
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.