From: Ian Campbell <ijc@hellion.org.uk>
To: Giuseppe Sacco <giuseppe@eppesuigoccas.homedns.org>,
638172@bugs.debian.org
Cc: xen-devel <xen-devel@lists.xensource.com>,
Ben Hutchings <ben@decadent.org.uk>
Subject: Re: Bug#638172: BUG: soft lockup - CPU#0 stuck for 61s! [qemu-dm:3205]
Date: Mon, 22 Aug 2011 10:00:11 +0100 [thread overview]
Message-ID: <1314003611.5010.400.camel@zakaz.uk.xensource.com> (raw)
In-Reply-To: <1313577856.13030.17.camel@scarafaggio>
@xen-devel:
Does this look familiar to anyone, this is (I expect, hopefully Giuseppe
will confirm) from Debian Squeeze which has a Xen 4.0.x with a PVops
dom0 kernel based on xen.git from last summer (e73f4955a821) with more
recent upstream longterm kernels (up to and including 2.6.32.41) merged
in. While it does seem to have the switch from level to edge triggered
interrupt the Debian kernel doesn't appear to have the switch to fasteoi
for pirqs (0672fb44a111 plus a few followups) -- could that be related
to this? (I'm not sure if that was a cleanup or a fix)
Might the tsc unstable message be relevant?
@Giuseppe:
Can you confirm the versions of the xen and qemu-dm packages which you
have got installed please.
Also I think it would be useful to see the guest configuration file and
details of the storage (filesystems, SCSI controllers etc) backing the
guest storage which you have got configured.
Full history of this report can be found at
http://bugs.debian.org/638172
Ian.
Can you also provide configuration details
On Wed, 2011-08-17 at 12:44 +0200, Giuseppe Sacco wrote:
> Package: linux-image-2.6.32-5-xen-686
> Version: 2.6.32-35
> Severity: important
>
> Hi,
> I am experiencing a few outages on a XEN server. Often I have to
> poweroff the server, but last time I found some information in syslog.
> Here it is:
>
> Aug 17 12:35:45 centrum kernel: [ 1424.037532] Clocksource tsc unstable (delta = -103103328 ns)
> Aug 17 12:35:45 centrum kernel: [ 1456.620463] BUG: soft lockup - CPU#0 stuck for 61s! [qemu-dm:3205]
> Aug 17 12:35:45 centrum kernel: [ 1456.620463] Modules linked in: xt_state xt_physdev iptable_filter tun cpufreq_userspace cpufreq_powersave cpufreq_c
> onservative cpufreq_stats dummy bridge stp xen_evtchn xenfs xt_tcpudp iptable_nat nf_nat nf_conntrack_ipv4 nf_conntrack nf_defrag_ipv4 ip_tables x_tab
> les xfs exportfs loop snd_hda_codec_atihdmi snd_hda_intel snd_hda_codec radeon snd_hwdep ttm snd_pcm snd_timer drm_kms_helper drm snd soundcore snd_pa
> ge_alloc i2c_algo_bit shpchp i2c_piix4 pcspkr k8temp pci_hotplug i2c_core evdev button ext3 jbd mbcache dm_mod aacraid 3w_9xxx 3w_xxxx raid10 raid456
> async_raid6_recov async_pq raid6_pq async_xor xor async_memcpy async_tx raid1 raid0 md_mod sata_nv sata_sil sata_via sd_mod crc_t10dif ata_generic ahc
> i pata_atiixp ohci_hcd libata processor ehci_hcd r8169 mii scsi_mod thermal usbcore nls_base thermal_sys acpi_processor [last unloaded: dummy]
> Aug 17 12:35:45 centrum kernel: [ 1456.620463]
> Aug 17 12:35:45 centrum kernel: [ 1456.620463] Pid: 3205, comm: qemu-dm Tainted: G W (2.6.32-5-xen-686 #1) MS-7368
> Aug 17 12:35:45 centrum kernel: [ 1456.620463] EIP: 0061:[<c1002227>] EFLAGS: 00200246 CPU: 0
> Aug 17 12:35:45 centrum kernel: [ 1456.620463] EIP is at hypercall_page+0x227/0x1001
> Aug 17 12:35:45 centrum kernel: [ 1456.620463] EAX: 00040000 EBX: 00000000 ECX: 00000000 EDX: ec8fa828
> Aug 17 12:35:45 centrum kernel: [ 1456.620463] ESI: ec8fa800 EDI: c24d9600 EBP: c27d4800 ESP: e4207d64
> Aug 17 12:35:45 centrum kernel: [ 1456.620463] DS: 007b ES: 007b FS: 00d8 GS: 00e0 SS: 0069
> Aug 17 12:35:45 centrum kernel: [ 1456.620463] CR0: 8005003b CR2: b7712200 CR3: 241f0000 CR4: 00000660
> Aug 17 12:35:45 centrum kernel: [ 1456.620463] DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3: 00000000
> Aug 17 12:35:45 centrum kernel: [ 1456.620463] DR6: ffff0ff0 DR7: 00000400
> Aug 17 12:35:45 centrum kernel: [ 1456.620463] Call Trace:
> Aug 17 12:35:45 centrum kernel: [ 1456.620463] [<c1006034>] ? xen_force_evtchn_callback+0xc/0x10
> Aug 17 12:35:45 centrum kernel: [ 1456.620463] [<c1006764>] ? check_events+0x8/0xc
> Aug 17 12:35:45 centrum kernel: [ 1456.620463] [<c1006723>] ? xen_irq_enable_direct_end+0x0/0x1
> Aug 17 12:35:45 centrum kernel: [ 1456.620463] [<ed93f457>] ? scsi_request_fn+0x440/0x47a [scsi_mod]
> Aug 17 12:35:45 centrum kernel: [ 1456.620463] [<c1132541>] ? __blk_run_queue+0x2e/0x5a
> Aug 17 12:35:45 centrum kernel: [ 1456.620463] [<c11325f3>] ? blk_run_queue+0x18/0x27
> Aug 17 12:35:45 centrum kernel: [ 1456.620463] [<ed93eaca>] ? scsi_run_queue+0x281/0x308 [scsi_mod]
> Aug 17 12:35:45 centrum kernel: [ 1456.620463] [<ed93f639>] ? scsi_next_command+0x25/0x2f [scsi_mod]
> Aug 17 12:35:45 centrum kernel: [ 1456.620463] [<ed93ffa1>] ? scsi_io_completion+0x383/0x3a4 [scsi_mod]
> Aug 17 12:35:45 centrum kernel: [ 1456.620463] [<ed93a723>] ? scsi_finish_command+0xaa/0xc2 [scsi_mod]
> Aug 17 12:35:45 centrum kernel: [ 1456.620463] [<c1135bb3>] ? blk_done_softirq+0x53/0x5f
> Aug 17 12:35:45 centrum kernel: [ 1456.620463] [<c103c8ea>] ? __do_softirq+0xaa/0x156
> Aug 17 12:35:45 centrum kernel: [ 1456.620463] [<c103c9c7>] ? do_softirq+0x31/0x3c
> Aug 17 12:35:45 centrum kernel: [ 1456.620463] [<c103caa1>] ? irq_exit+0x26/0x58
> Aug 17 12:35:45 centrum kernel: [ 1456.620463] [<c1199be6>] ? xen_evtchn_do_upcall+0x22/0x2c
> Aug 17 12:35:45 centrum kernel: [ 1456.620463] [<c1009b3f>] ? xen_do_upcall+0x7/0xc
> Aug 17 12:35:45 centrum kernel: [ 1456.620463] [<c1002407>] ? hypercall_page+0x407/0x1001
> Aug 17 12:35:45 centrum kernel: [ 1456.620463] [<eda10015>] ? HYPERVISOR_event_channel_op+0x15/0x4c [xen_evtchn]
> Aug 17 12:35:45 centrum kernel: [ 1456.620463] [<c10937c6>] ? __alloc_pages_nodemask+0xf3/0x4d9
> Aug 17 12:35:45 centrum kernel: [ 1456.620463] EIP is at hypercall_page+0x227/0x1001
> Aug 17 12:35:45 centrum kernel: [ 1456.620463] EAX: 00040000 EBX: 00000000 ECX: 00000000 EDX: ec8fa828
> Aug 17 12:35:45 centrum kernel: [ 1456.620463] ESI: ec8fa800 EDI: c24d9600 EBP: c27d4800 ESP: e4207d64
> Aug 17 12:35:45 centrum kernel: [ 1456.620463] DS: 007b ES: 007b FS: 00d8 GS: 00e0 SS: 0069
> Aug 17 12:35:45 centrum kernel: [ 1456.620463] CR0: 8005003b CR2: b7712200 CR3: 241f0000 CR4: 00000660
> Aug 17 12:35:45 centrum kernel: [ 1456.620463] DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3: 00000000
> Aug 17 12:35:45 centrum kernel: [ 1456.620463] DR6: ffff0ff0 DR7: 00000400
> Aug 17 12:35:45 centrum kernel: [ 1456.620463] Call Trace:
> Aug 17 12:35:45 centrum kernel: [ 1456.620463] [<c1006034>] ? xen_force_evtchn_callback+0xc/0x10
> Aug 17 12:35:45 centrum kernel: [ 1456.620463] [<c1006764>] ? check_events+0x8/0xc
> Aug 17 12:35:45 centrum kernel: [ 1456.620463] [<c1006723>] ? xen_irq_enable_direct_end+0x0/0x1
> Aug 17 12:35:45 centrum kernel: [ 1456.620463] [<ed93f457>] ? scsi_request_fn+0x440/0x47a [scsi_mod]
> Aug 17 12:35:45 centrum kernel: [ 1456.620463] [<c1132541>] ? __blk_run_queue+0x2e/0x5a
> Aug 17 12:35:45 centrum kernel: [ 1456.620463] [<c11325f3>] ? blk_run_queue+0x18/0x27
> Aug 17 12:35:45 centrum kernel: [ 1456.620463] [<ed93eaca>] ? scsi_run_queue+0x281/0x308 [scsi_mod]
> Aug 17 12:35:45 centrum kernel: [ 1456.620463] [<ed93f639>] ? scsi_next_command+0x25/0x2f [scsi_mod]
> Aug 17 12:35:45 centrum kernel: [ 1456.620463] [<ed93ffa1>] ? scsi_io_completion+0x383/0x3a4 [scsi_mod]
> Aug 17 12:35:45 centrum kernel: [ 1456.620463] [<ed93a723>] ? scsi_finish_command+0xaa/0xc2 [scsi_mod]
> Aug 17 12:35:45 centrum kernel: [ 1456.620463] [<c1135bb3>] ? blk_done_softirq+0x53/0x5f
> Aug 17 12:35:45 centrum kernel: [ 1456.620463] [<c103c8ea>] ? __do_softirq+0xaa/0x156
> Aug 17 12:35:45 centrum kernel: [ 1456.620463] [<c103c9c7>] ? do_softirq+0x31/0x3c
> Aug 17 12:35:45 centrum kernel: [ 1456.620463] [<c103caa1>] ? irq_exit+0x26/0x58
> Aug 17 12:35:45 centrum kernel: [ 1456.620463] [<c1199be6>] ? xen_evtchn_do_upcall+0x22/0x2c
> Aug 17 12:35:45 centrum kernel: [ 1456.620463] [<c1009b3f>] ? xen_do_upcall+0x7/0xc
> Aug 17 12:35:45 centrum kernel: [ 1456.620463] [<c1002407>] ? hypercall_page+0x407/0x1001
> Aug 17 12:35:45 centrum kernel: [ 1456.620463] [<eda10015>] ? HYPERVISOR_event_channel_op+0x15/0x4c [xen_evtchn]
> Aug 17 12:35:45 centrum kernel: [ 1456.620463] [<c10937c6>] ? __alloc_pages_nodemask+0xf3/0x4d9
> Aug 17 12:35:45 centrum kernel: [ 1456.620463] [<eda10484>] ? evtchn_ioctl+0x22e/0x28c [xen_evtchn]
> Aug 17 12:35:45 centrum kernel: [ 1456.620463] [<eda10256>] ? evtchn_ioctl+0x0/0x28c [xen_evtchn]
> Aug 17 12:35:45 centrum kernel: [ 1456.620463] [<c10c6520>] ? vfs_ioctl+0x1c/0x5f
> Aug 17 12:35:45 centrum kernel: [ 1456.620463] [<c10c6ab4>] ? do_vfs_ioctl+0x4aa/0x4e5
> Aug 17 12:35:45 centrum kernel: [ 1456.620463] [<c10bb8e6>] ? fsnotify_modify+0x5a/0x61
> Aug 17 12:35:45 centrum kernel: [ 1456.620463] [<eda104e2>] ? evtchn_write+0x0/0xda [xen_evtchn]
> Aug 17 12:35:45 centrum kernel: [ 1456.620463] [<c10bc4c4>] ? vfs_write+0x9e/0xd6
> Aug 17 12:35:45 centrum kernel: [ 1456.620463] [<c10c6b30>] ? sys_ioctl+0x41/0x58
> Aug 17 12:35:45 centrum kernel: [ 1456.620463] [<c1008f7c>] ? syscall_call+0x7/0xb
>
> Bye,
> Giuseppe
>
>
>
>
--
Ian Campbell
Current Noise: Converge - For You (Live)
(null cookie; hope that's ok)
next parent reply other threads:[~2011-08-22 9:00 UTC|newest]
Thread overview: 12+ messages / expand[flat|nested] mbox.gz Atom feed top
[not found] <1313577856.13030.17.camel@scarafaggio>
2011-08-22 9:00 ` Ian Campbell [this message]
2011-08-22 9:49 ` Bug#638172: BUG: soft lockup - CPU#0 stuck for 61s! [qemu-dm:3205] Giuseppe Sacco
2011-08-24 20:24 ` Konrad Rzeszutek Wilk
2011-08-24 21:24 ` Ian Campbell
2011-08-25 6:52 ` Bug#638172: [Xen-devel] " Giuseppe Sacco
2011-08-25 6:56 ` Ian Campbell
2011-08-25 7:23 ` Bug#638172: [Xen-devel] " Giuseppe Sacco
2011-08-26 7:25 ` Bug#638172: " Ian Campbell
2011-08-26 8:28 ` Giuseppe Sacco
2011-08-26 8:31 ` Ian Campbell
2011-10-18 10:54 ` Ian Campbell
2011-10-18 11:00 ` Giuseppe Sacco
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=1314003611.5010.400.camel@zakaz.uk.xensource.com \
--to=ijc@hellion.org.uk \
--cc=638172@bugs.debian.org \
--cc=ben@decadent.org.uk \
--cc=giuseppe@eppesuigoccas.homedns.org \
--cc=xen-devel@lists.xensource.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.