* [RFC] NOHZ: fix nohz on cpu unplug
@ 2009-01-30 16:29 Christian Borntraeger
2009-02-02 9:58 ` [PATCH/RFC] " Christian Borntraeger
0 siblings, 1 reply; 12+ messages in thread
From: Christian Borntraeger @ 2009-01-30 16:29 UTC (permalink / raw
To: Ingo Molnar, Thomas Gleixner
Cc: linux-kernel, Heiko Carstens, Martin Schwidefsky
Ingo, Thomas,
After using cpu unplug I have seen one cpu with enabled ticks on s390,
even on a idle systems. It turned out that nohz.cpu_mask is not
updated on cpu unplug.
In select_nohz_load_balancer we check if the system is completely
idle to turn of load balancing. We compare the weights of
cpu_online_map with nohz.cpu_mask.
Since cpu_online_map is updated on cpu unplug, but nohz.cpu_mask is
not, the check fails and the scheduler believes that we need an
"idle load balancer" even on a fully idle system. Since the ilb
cpu does not deactivate the timer tick this "breaks" NOHZ.
One possible fix is to only add the cpu to the nohz.cpu_mask,
if the cpu is active.
The alternative is to add nohz.cpu_mask in the cpu hotplug path.
This patch seems to work on s390 (Shortly tested with kvm guests
with cpu hotplug and kvm_stat on the host), but I want to hear your
opinion.
Signed-off-by: Christian Borntraeger <borntraeger@de.ibm.com>
---
kernel/sched.c | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
Index: kvm/kernel/sched.c
===================================================================
--- kvm.orig/kernel/sched.c
+++ kvm/kernel/sched.c
@@ -3880,7 +3880,8 @@ int select_nohz_load_balancer(int stop_t
int cpu = smp_processor_id();
if (stop_tick) {
- cpumask_set_cpu(cpu, nohz.cpu_mask);
+ if (cpu_active(cpu))
+ cpumask_set_cpu(cpu, nohz.cpu_mask);
cpu_rq(cpu)->in_nohz_recently = 1;
/*
^ permalink raw reply [flat|nested] 12+ messages in thread
* [PATCH/RFC] NOHZ: fix nohz on cpu unplug
2009-01-30 16:29 [RFC] NOHZ: fix nohz on cpu unplug Christian Borntraeger
@ 2009-02-02 9:58 ` Christian Borntraeger
2009-02-02 12:23 ` [PATCH v2] " Christian Borntraeger
0 siblings, 1 reply; 12+ messages in thread
From: Christian Borntraeger @ 2009-02-02 9:58 UTC (permalink / raw
To: Ingo Molnar, Thomas Gleixner
Cc: linux-kernel, Heiko Carstens, Martin Schwidefsky, Siddha,
Suresh B
This is a reminder mail. I still dont know if my fix is correct, although
it seems to work..
I also CCed: Siddha, Suresh B <suresh.b.siddha@intel.com>
> Ingo, Thomas,
>
> After using cpu unplug I have seen one cpu with enabled ticks on s390,
> even on a idle systems. It turned out that nohz.cpu_mask is not
> updated on cpu unplug.
> In select_nohz_load_balancer we check if the system is completely
> idle to turn of load balancing. We compare the weights of
> cpu_online_map with nohz.cpu_mask.
> Since cpu_online_map is updated on cpu unplug, but nohz.cpu_mask is
> not, the check fails and the scheduler believes that we need an
> "idle load balancer" even on a fully idle system. Since the ilb
> cpu does not deactivate the timer tick this "breaks" NOHZ.
>
> One possible fix is to only add the cpu to the nohz.cpu_mask,
> if the cpu is active.
>
> The alternative is to add nohz.cpu_mask in the cpu hotplug path.
>
> This patch seems to work on s390 (Shortly tested with kvm guests
> with cpu hotplug and kvm_stat on the host), but I want to hear your
> opinion.
>
> Signed-off-by: Christian Borntraeger <borntraeger@de.ibm.com>
> ---
> kernel/sched.c | 2 +-
> 1 file changed, 1 insertion(+), 1 deletion(-)
>
> Index: kvm/kernel/sched.c
> ===================================================================
> --- kvm.orig/kernel/sched.c
> +++ kvm/kernel/sched.c
> @@ -3880,7 +3880,8 @@ int select_nohz_load_balancer(int stop_t
> int cpu = smp_processor_id();
>
> if (stop_tick) {
> - cpumask_set_cpu(cpu, nohz.cpu_mask);
> + if (cpu_active(cpu))
> + cpumask_set_cpu(cpu, nohz.cpu_mask);
> cpu_rq(cpu)->in_nohz_recently = 1;
>
> /*
^ permalink raw reply [flat|nested] 12+ messages in thread
* [PATCH v2] NOHZ: fix nohz on cpu unplug
2009-02-02 9:58 ` [PATCH/RFC] " Christian Borntraeger
@ 2009-02-02 12:23 ` Christian Borntraeger
2009-02-02 23:57 ` Suresh Siddha
0 siblings, 1 reply; 12+ messages in thread
From: Christian Borntraeger @ 2009-02-02 12:23 UTC (permalink / raw
To: Ingo Molnar, Thomas Gleixner, Suresh B
Cc: linux-kernel, Heiko Carstens, Martin Schwidefsky
After some testing, I think this patch is better:
[PATCH] NOHZ: fix nohz on cpu unplug
From: Christian Borntraeger <borntraeger@de.ibm.com>
After using cpu unplug I have seen one cpu with full ticks, even
on a idle systems. It turns out that nohz.cpu_mask is not updated on
cpu unplug.
In select_nohz_load_balancer we check if the system is completely
idle to turn of load balancing. We compare cpu_online_map with
nohz.cpu_mask.
Since cpu_online_map is updated on cpu unplug, but nohz.cpu_mask is
not, the check fails and the scheduler believes that we need an
"idle load balancer" even on a fully idle system. Since the ilb
cpu does not deactivate the timer tick this breaks NOHZ.
This patch clear the nohz bits in the migration_call, a function that
is already called by the cpu hotplug notifier.
Opinions?
Signed-off-by: Christian Borntraeger <borntraeger@de.ibm.com>
---
kernel/sched.c | 1 +
1 file changed, 1 insertion(+)
Index: kvm/kernel/sched.c
===================================================================
--- kvm.orig/kernel/sched.c
+++ kvm/kernel/sched.c
@@ -6696,6 +6696,7 @@ migration_call(struct notifier_block *nf
rq->idle->sched_class = &idle_sched_class;
migrate_dead_tasks(cpu);
spin_unlock_irq(&rq->lock);
+ cpumask_clear_cpu(cpu, nohz.cpu_mask);
cpuset_unlock();
migrate_nr_uninterruptible(rq);
BUG_ON(rq->nr_running != 0);
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [PATCH v2] NOHZ: fix nohz on cpu unplug
2009-02-02 12:23 ` [PATCH v2] " Christian Borntraeger
@ 2009-02-02 23:57 ` Suresh Siddha
2009-02-03 8:48 ` Christian Borntraeger
0 siblings, 1 reply; 12+ messages in thread
From: Suresh Siddha @ 2009-02-02 23:57 UTC (permalink / raw
To: Christian Borntraeger
Cc: Ingo Molnar, Thomas Gleixner, linux-kernel@vger.kernel.org,
Heiko Carstens, Martin Schwidefsky
On Mon, 2009-02-02 at 04:23 -0800, Christian Borntraeger wrote:
> After some testing, I think this patch is better:
> [PATCH] NOHZ: fix nohz on cpu unplug
>
> From: Christian Borntraeger <borntraeger@de.ibm.com>
>
> After using cpu unplug I have seen one cpu with full ticks, even
> on a idle systems. It turns out that nohz.cpu_mask is not updated on
> cpu unplug.
> In select_nohz_load_balancer we check if the system is completely
> idle to turn of load balancing. We compare cpu_online_map with
> nohz.cpu_mask.
> Since cpu_online_map is updated on cpu unplug, but nohz.cpu_mask is
> not, the check fails and the scheduler believes that we need an
> "idle load balancer" even on a fully idle system. Since the ilb
> cpu does not deactivate the timer tick this breaks NOHZ.
>
> This patch clear the nohz bits in the migration_call, a function that
> is already called by the cpu hotplug notifier.
>
> Opinions?
Hi,
select_nohz_load_balancer() already takes care of one cpu offline
scenario. It missed one more scenario leading to the issue you
explained. We should take care of this in select_nohz_load_balancer().
Does this patch fix the issue? Thanks.
---
diff --git a/kernel/sched.c b/kernel/sched.c
index 96439a4..8a0419b 100644
--- a/kernel/sched.c
+++ b/kernel/sched.c
@@ -4026,19 +4026,24 @@ int select_nohz_load_balancer(int stop_tick)
int cpu = smp_processor_id();
if (stop_tick) {
- cpumask_set_cpu(cpu, nohz.cpu_mask);
cpu_rq(cpu)->in_nohz_recently = 1;
- /*
- * If we are going offline and still the leader, give up!
- */
- if (!cpu_active(cpu) &&
- atomic_read(&nohz.load_balancer) == cpu) {
+ if (!cpu_active(cpu) {
+ if (atomic_read(&nohz.load_balancer) != cpu)
+ return 0;
+
+ /*
+ * If we are going offline and still the leader,
+ * give up!
+ */
if (atomic_cmpxchg(&nohz.load_balancer, cpu, -1) != cpu)
BUG();
+
return 0;
}
+ cpumask_set_cpu(cpu, nohz.cpu_mask);
+
/* time for ilb owner also to sleep */
if (cpumask_weight(nohz.cpu_mask) == num_online_cpus()) {
if (atomic_read(&nohz.load_balancer) == cpu)
^ permalink raw reply related [flat|nested] 12+ messages in thread
* Re: [PATCH v2] NOHZ: fix nohz on cpu unplug
2009-02-02 23:57 ` Suresh Siddha
@ 2009-02-03 8:48 ` Christian Borntraeger
2009-02-03 12:28 ` Ingo Molnar
0 siblings, 1 reply; 12+ messages in thread
From: Christian Borntraeger @ 2009-02-03 8:48 UTC (permalink / raw
To: Suresh Siddha
Cc: Ingo Molnar, Thomas Gleixner, linux-kernel@vger.kernel.org,
Heiko Carstens, Martin Schwidefsky
Am Tuesday 03 February 2009 00:57:03 schrieb Suresh Siddha:
> Does this patch fix the issue? Thanks.
I fixed up a compile error. (See below). After that, the patch does indeed fix
the problem.
Tested-by: Christian Borntraeger <borntraeger@de.ibm.com>
> diff --git a/kernel/sched.c b/kernel/sched.c
> index 96439a4..8a0419b 100644
> --- a/kernel/sched.c
> +++ b/kernel/sched.c
> @@ -4026,19 +4026,24 @@ int select_nohz_load_balancer(int stop_tick)
> int cpu = smp_processor_id();
>
> if (stop_tick) {
> - cpumask_set_cpu(cpu, nohz.cpu_mask);
> cpu_rq(cpu)->in_nohz_recently = 1;
>
> - /*
> - * If we are going offline and still the leader, give up!
> - */
> - if (!cpu_active(cpu) &&
> - atomic_read(&nohz.load_balancer) == cpu) {
> + if (!cpu_active(cpu) {
That does not compile...I used the following:
if (!cpu_active(cpu)) {
> + if (atomic_read(&nohz.load_balancer) != cpu)
> + return 0;
> +
> + /*
> + * If we are going offline and still the leader,
> + * give up!
> + */
> if (atomic_cmpxchg(&nohz.load_balancer, cpu, -1) != cpu)
> BUG();
> +
> return 0;
> }
>
> + cpumask_set_cpu(cpu, nohz.cpu_mask);
> +
> /* time for ilb owner also to sleep */
> if (cpumask_weight(nohz.cpu_mask) == num_online_cpus()) {
> if (atomic_read(&nohz.load_balancer) == cpu)
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [PATCH v2] NOHZ: fix nohz on cpu unplug
2009-02-03 8:48 ` Christian Borntraeger
@ 2009-02-03 12:28 ` Ingo Molnar
2009-02-04 6:19 ` Christian Borntraeger
0 siblings, 1 reply; 12+ messages in thread
From: Ingo Molnar @ 2009-02-03 12:28 UTC (permalink / raw
To: Christian Borntraeger
Cc: Suresh Siddha, Thomas Gleixner, linux-kernel@vger.kernel.org,
Heiko Carstens, Martin Schwidefsky
* Christian Borntraeger <borntraeger@de.ibm.com> wrote:
> Am Tuesday 03 February 2009 00:57:03 schrieb Suresh Siddha:
> > Does this patch fix the issue? Thanks.
>
> I fixed up a compile error. (See below). After that, the patch does indeed fix
> the problem.
>
> Tested-by: Christian Borntraeger <borntraeger@de.ibm.com>
>
>
> > diff --git a/kernel/sched.c b/kernel/sched.c
> > index 96439a4..8a0419b 100644
> > --- a/kernel/sched.c
> > +++ b/kernel/sched.c
> > @@ -4026,19 +4026,24 @@ int select_nohz_load_balancer(int stop_tick)
> > int cpu = smp_processor_id();
> >
> > if (stop_tick) {
> > - cpumask_set_cpu(cpu, nohz.cpu_mask);
> > cpu_rq(cpu)->in_nohz_recently = 1;
> >
> > - /*
> > - * If we are going offline and still the leader, give up!
> > - */
> > - if (!cpu_active(cpu) &&
> > - atomic_read(&nohz.load_balancer) == cpu) {
> > + if (!cpu_active(cpu) {
>
> That does not compile...I used the following:
> if (!cpu_active(cpu)) {
Could you please resubmit an updated patch?
Ingo
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [PATCH v2] NOHZ: fix nohz on cpu unplug
2009-02-03 12:28 ` Ingo Molnar
@ 2009-02-04 6:19 ` Christian Borntraeger
2009-02-04 19:59 ` Suresh Siddha
0 siblings, 1 reply; 12+ messages in thread
From: Christian Borntraeger @ 2009-02-04 6:19 UTC (permalink / raw
To: Suresh Siddha
Cc: Ingo Molnar, Thomas Gleixner, linux-kernel@vger.kernel.org,
Heiko Carstens, Martin Schwidefsky
Am Tuesday 03 February 2009 13:28:47 schrieb Ingo Molnar:
> > > + if (!cpu_active(cpu) {
> >
> > That does not compile...I used the following:
> > if (!cpu_active(cpu)) {
>
> Could you please resubmit an updated patch?
Suresh,
can you send the updated patch? After all, its your work and we need your
Signed-off-by anyway.
You can add my
Reported-by: Christian Borntraeger <borntraeger@de.ibm.com>
Tested-by: Christian Borntraeger <borntraeger@de.ibm.com>
Thanks
Christian
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [PATCH v2] NOHZ: fix nohz on cpu unplug
2009-02-04 6:19 ` Christian Borntraeger
@ 2009-02-04 19:59 ` Suresh Siddha
2009-02-04 21:32 ` Ingo Molnar
0 siblings, 1 reply; 12+ messages in thread
From: Suresh Siddha @ 2009-02-04 19:59 UTC (permalink / raw
To: Christian Borntraeger
Cc: Ingo Molnar, Thomas Gleixner, linux-kernel@vger.kernel.org,
Heiko Carstens, Martin Schwidefsky
On Tue, 2009-02-03 at 22:19 -0800, Christian Borntraeger wrote:
> Am Tuesday 03 February 2009 13:28:47 schrieb Ingo Molnar:
> > > > + if (!cpu_active(cpu) {
> > >
> > > That does not compile...I used the following:
> > > if (!cpu_active(cpu)) {
> >
> > Could you please resubmit an updated patch?
>
> Suresh,
>
> can you send the updated patch? After all, its your work and we need your
> Signed-off-by anyway.
>
> You can add my
> Reported-by: Christian Borntraeger <borntraeger@de.ibm.com>
> Tested-by: Christian Borntraeger <borntraeger@de.ibm.com>
Thanks Christian for testing the fix. Ingo, appended is the updated patch. Thanks.
---
From: Suresh Siddha <suresh.b.siddha@intel.com>
Subject: sched: fix nohz load balancer on cpu offline
Christian Borntraeger reports:
> After a logical cpu offline, even on a complete idle system, there
> is one cpu with full ticks. It turns out that nohz.cpu_mask has the
> the offlined cpu still set.
>
> In select_nohz_load_balancer() we check if the system is completely
> idle to turn of load balancing. We compare cpu_online_map with
> nohz.cpu_mask. Since cpu_online_map is updated on cpu unplug,
> but nohz.cpu_mask is not, the check fails and the scheduler believes
that
> we need an "idle load balancer" even on a fully idle system. Since the
ilb
> cpu does not deactivate the timer tick this breaks NOHZ.
Fix the select_nohz_load_balancer() to not set the nohz.cpu_mask
while a cpu is going offline.
Reported-by: Christian Borntraeger <borntraeger@de.ibm.com>
Signed-off-by: Suresh Siddha <suresh.b.siddha@intel.com>
Tested-by: Christian Borntraeger <borntraeger@de.ibm.com>
---
diff --git a/kernel/sched.c b/kernel/sched.c
index 96439a4..8a0419b 100644
--- a/kernel/sched.c
+++ b/kernel/sched.c
@@ -4026,19 +4026,24 @@ int select_nohz_load_balancer(int stop_tick)
int cpu = smp_processor_id();
if (stop_tick) {
- cpumask_set_cpu(cpu, nohz.cpu_mask);
cpu_rq(cpu)->in_nohz_recently = 1;
- /*
- * If we are going offline and still the leader, give up!
- */
- if (!cpu_active(cpu) &&
- atomic_read(&nohz.load_balancer) == cpu) {
+ if (!cpu_active(cpu)) {
+ if (atomic_read(&nohz.load_balancer) != cpu)
+ return 0;
+
+ /*
+ * If we are going offline and still the leader,
+ * give up!
+ */
if (atomic_cmpxchg(&nohz.load_balancer, cpu, -1) != cpu)
BUG();
+
return 0;
}
+ cpumask_set_cpu(cpu, nohz.cpu_mask);
+
/* time for ilb owner also to sleep */
if (cpumask_weight(nohz.cpu_mask) == num_online_cpus()) {
if (atomic_read(&nohz.load_balancer) == cpu)
^ permalink raw reply related [flat|nested] 12+ messages in thread
* Re: [PATCH v2] NOHZ: fix nohz on cpu unplug
2009-02-04 19:59 ` Suresh Siddha
@ 2009-02-04 21:32 ` Ingo Molnar
2009-02-05 19:54 ` Mark Lord
0 siblings, 1 reply; 12+ messages in thread
From: Ingo Molnar @ 2009-02-04 21:32 UTC (permalink / raw
To: Suresh Siddha
Cc: Christian Borntraeger, Thomas Gleixner,
linux-kernel@vger.kernel.org, Heiko Carstens, Martin Schwidefsky
* Suresh Siddha <suresh.b.siddha@intel.com> wrote:
> On Tue, 2009-02-03 at 22:19 -0800, Christian Borntraeger wrote:
> > Am Tuesday 03 February 2009 13:28:47 schrieb Ingo Molnar:
> > > > > + if (!cpu_active(cpu) {
> > > >
> > > > That does not compile...I used the following:
> > > > if (!cpu_active(cpu)) {
> > >
> > > Could you please resubmit an updated patch?
> >
> > Suresh,
> >
> > can you send the updated patch? After all, its your work and we need your
> > Signed-off-by anyway.
> >
> > You can add my
> > Reported-by: Christian Borntraeger <borntraeger@de.ibm.com>
> > Tested-by: Christian Borntraeger <borntraeger@de.ibm.com>
>
> Thanks Christian for testing the fix. Ingo, appended is the updated patch.
> Thanks.
applied to tip:sched/urgent, thanks guys!
Ingo
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [PATCH v2] NOHZ: fix nohz on cpu unplug
2009-02-04 21:32 ` Ingo Molnar
@ 2009-02-05 19:54 ` Mark Lord
2009-02-06 2:33 ` Suresh Siddha
0 siblings, 1 reply; 12+ messages in thread
From: Mark Lord @ 2009-02-05 19:54 UTC (permalink / raw
To: Ingo Molnar
Cc: Suresh Siddha, Christian Borntraeger, Thomas Gleixner,
linux-kernel@vger.kernel.org, Heiko Carstens, Martin Schwidefsky
Ingo Molnar wrote:
> * Suresh Siddha <suresh.b.siddha@intel.com> wrote:
>
>> On Tue, 2009-02-03 at 22:19 -0800, Christian Borntraeger wrote:
>>> Am Tuesday 03 February 2009 13:28:47 schrieb Ingo Molnar:
>>>>>> + if (!cpu_active(cpu) {
>>>>> That does not compile...I used the following:
>>>>> if (!cpu_active(cpu)) {
>>>> Could you please resubmit an updated patch?
>>> Suresh,
>>>
>>> can you send the updated patch? After all, its your work and we need your
>>> Signed-off-by anyway.
>>>
>>> You can add my
>>> Reported-by: Christian Borntraeger <borntraeger@de.ibm.com>
>>> Tested-by: Christian Borntraeger <borntraeger@de.ibm.com>
>> Thanks Christian for testing the fix. Ingo, appended is the updated patch.
>> Thanks.
>
> applied to tip:sched/urgent, thanks guys!
>
> Ingo
..
How far back in (kernel release) time does this problem exist?
Candidate for -stable ?
Cheers
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [PATCH v2] NOHZ: fix nohz on cpu unplug
2009-02-05 19:54 ` Mark Lord
@ 2009-02-06 2:33 ` Suresh Siddha
2009-02-06 14:52 ` Mark Lord
0 siblings, 1 reply; 12+ messages in thread
From: Suresh Siddha @ 2009-02-06 2:33 UTC (permalink / raw
To: Mark Lord
Cc: Ingo Molnar, Christian Borntraeger, Thomas Gleixner,
linux-kernel@vger.kernel.org, Heiko Carstens, Martin Schwidefsky
On Thu, 2009-02-05 at 11:54 -0800, Mark Lord wrote:
> How far back in (kernel release) time does this problem exist?
> Candidate for -stable ?
Problem is present for a while now. But I don't think this is a common
case scenario (as the issue happens only for the duration when we leave
a cpu offline, and it should get fixed the moment that logical cpu is
back online).
thanks,
suresh
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [PATCH v2] NOHZ: fix nohz on cpu unplug
2009-02-06 2:33 ` Suresh Siddha
@ 2009-02-06 14:52 ` Mark Lord
0 siblings, 0 replies; 12+ messages in thread
From: Mark Lord @ 2009-02-06 14:52 UTC (permalink / raw
To: Suresh Siddha
Cc: Ingo Molnar, Christian Borntraeger, Thomas Gleixner,
linux-kernel@vger.kernel.org, Heiko Carstens, Martin Schwidefsky
Suresh Siddha wrote:
> On Thu, 2009-02-05 at 11:54 -0800, Mark Lord wrote:
>> How far back in (kernel release) time does this problem exist?
>> Candidate for -stable ?
>
> Problem is present for a while now. But I don't think this is a common
> case scenario (as the issue happens only for the duration when we leave
> a cpu offline, and it should get fixed the moment that logical cpu is
> back online).
..
There is an existing bug (for some time now) in the kernel shutdown
for multi-CPUs. Once in a while, perhaps every 20-30 halts,
the kernel fails to power-off the machine.
I've seen this problem here since 2.6.18 or so, on multiple different
machines with Core2duo and Core2quad processors.
It comes and goes, depending upon the kernel version and exact .config
that is used. Any attempt to instrument it generally changes the race
conditions enough that it stops happening.
I'm just wondering if this bug might explain some of that.
Cheers
^ permalink raw reply [flat|nested] 12+ messages in thread
end of thread, other threads:[~2009-02-06 14:52 UTC | newest]
Thread overview: 12+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2009-01-30 16:29 [RFC] NOHZ: fix nohz on cpu unplug Christian Borntraeger
2009-02-02 9:58 ` [PATCH/RFC] " Christian Borntraeger
2009-02-02 12:23 ` [PATCH v2] " Christian Borntraeger
2009-02-02 23:57 ` Suresh Siddha
2009-02-03 8:48 ` Christian Borntraeger
2009-02-03 12:28 ` Ingo Molnar
2009-02-04 6:19 ` Christian Borntraeger
2009-02-04 19:59 ` Suresh Siddha
2009-02-04 21:32 ` Ingo Molnar
2009-02-05 19:54 ` Mark Lord
2009-02-06 2:33 ` Suresh Siddha
2009-02-06 14:52 ` Mark Lord
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).