On Sun, 2009-01-18 at 11:21 +0200, Avi Kivity wrote:
> If the host load is low enough, it may be worthwhile to repeat with 
> -no-kvm.  It's significantly different from kvm (much higher cpu load, 
> and less tasks involved), but if the problem recurs, we know it's a pure 
> sched issue.

Okay I repeated the test as best I could with the standard 2.6.28 kernel
and "-no-kvm" added to the command lines of all the guests. I couldn't
repeat exactly the same test due to some problems with the guests.

- Two of the XP guests were taking a looong time to complete their
  startup and get into idle mode. I waited ~30 minutes for them to
  settle, but they were still running with 100% CPU load, so I shut them
  down.
- The two linux SMP guests would not boot unless I added "nosmp" to the
  kernel command line in grub within the guest. Screenshots of where the
  two guests got stuck booting are attached.

Here are the results of the ping test, FWIW:

--- hermes-old.wumi.org.au ping statistics ---
900 packets transmitted, 900 received, 0% packet loss, time 903042ms
rtt min/avg/max/mdev = 0.388/2.389/73.171/3.685 ms

There's probably a few too many things different with this test compared
to the previous ones, but I guess it does point to the problem being an
interaction between both the scheduler and kvm. It may also be that it
requires smp guests to be running to trigger.

Regards,
Kevin.