* Wiki docs on counting and tracing KVM perf events
@ 2010-05-13 14:35 Stefan Hajnoczi
2010-05-13 14:57 ` Avi Kivity
0 siblings, 1 reply; 12+ messages in thread
From: Stefan Hajnoczi @ 2010-05-13 14:35 UTC (permalink / raw
To: kvm
How to count and trace KVM perf events:
http://www.linux-kvm.org/page/Perf_events
I want to draw attention to this because traditional kvm_stat and
kvm_trace use has been moving over to the debugfs based tracing
mechanisms. Perhaps we can flesh out documentation and examples of
common perf event usage.
Stefan
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: Wiki docs on counting and tracing KVM perf events
2010-05-13 14:35 Wiki docs on counting and tracing KVM perf events Stefan Hajnoczi
@ 2010-05-13 14:57 ` Avi Kivity
2010-05-20 8:15 ` Jes Sorensen
0 siblings, 1 reply; 12+ messages in thread
From: Avi Kivity @ 2010-05-13 14:57 UTC (permalink / raw
To: Stefan Hajnoczi; +Cc: kvm
On 05/13/2010 05:35 PM, Stefan Hajnoczi wrote:
> How to count and trace KVM perf events:
>
> http://www.linux-kvm.org/page/Perf_events
>
> I want to draw attention to this because traditional kvm_stat and
> kvm_trace use has been moving over to the debugfs based tracing
> mechanisms. Perhaps we can flesh out documentation and examples of
> common perf event usage.
>
>
Two things are missing to make this really useful:
- a continuously updating difference mode like kvm_stat
- subevents; for example kvm:kvm_exit is an aggregate of all exit types
that can be split using filters to show individual exit reason statistics
The plan is to eventually remove kvm_stat based performance monitoring
in favour of perf events.
--
error compiling committee.c: too many arguments to function
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: Wiki docs on counting and tracing KVM perf events
2010-05-13 14:57 ` Avi Kivity
@ 2010-05-20 8:15 ` Jes Sorensen
2010-05-20 8:19 ` Avi Kivity
0 siblings, 1 reply; 12+ messages in thread
From: Jes Sorensen @ 2010-05-20 8:15 UTC (permalink / raw
To: Avi Kivity; +Cc: Stefan Hajnoczi, kvm
On 05/13/10 16:57, Avi Kivity wrote:
> On 05/13/2010 05:35 PM, Stefan Hajnoczi wrote:
>> How to count and trace KVM perf events:
>>
>> http://www.linux-kvm.org/page/Perf_events
>>
>> I want to draw attention to this because traditional kvm_stat and
>> kvm_trace use has been moving over to the debugfs based tracing
>> mechanisms. Perhaps we can flesh out documentation and examples of
>> common perf event usage.
>>
>>
>
> Two things are missing to make this really useful:
>
> - a continuously updating difference mode like kvm_stat
> - subevents; for example kvm:kvm_exit is an aggregate of all exit types
> that can be split using filters to show individual exit reason statistics
Third missing item, which I find really useful:
- run once spit out raw counters
For some operations, like file system benchmarking, it is useful to
sample the counters before and after and then divide the raw number of
events by the number of IOPS performed by the benchmark. If perf spits
out events/sec it's kinda hard to get this.
Cheers,
Jes
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: Wiki docs on counting and tracing KVM perf events
2010-05-20 8:15 ` Jes Sorensen
@ 2010-05-20 8:19 ` Avi Kivity
2010-05-20 8:24 ` Jes Sorensen
0 siblings, 1 reply; 12+ messages in thread
From: Avi Kivity @ 2010-05-20 8:19 UTC (permalink / raw
To: Jes Sorensen; +Cc: Stefan Hajnoczi, kvm
On 05/20/2010 11:15 AM, Jes Sorensen wrote:
> On 05/13/10 16:57, Avi Kivity wrote:
>
>> On 05/13/2010 05:35 PM, Stefan Hajnoczi wrote:
>>
>>> How to count and trace KVM perf events:
>>>
>>> http://www.linux-kvm.org/page/Perf_events
>>>
>>> I want to draw attention to this because traditional kvm_stat and
>>> kvm_trace use has been moving over to the debugfs based tracing
>>> mechanisms. Perhaps we can flesh out documentation and examples of
>>> common perf event usage.
>>>
>>>
>>>
>> Two things are missing to make this really useful:
>>
>> - a continuously updating difference mode like kvm_stat
>> - subevents; for example kvm:kvm_exit is an aggregate of all exit types
>> that can be split using filters to show individual exit reason statistics
>>
> Third missing item, which I find really useful:
> - run once spit out raw counters
>
> For some operations, like file system benchmarking, it is useful to
> sample the counters before and after and then divide the raw number of
> events by the number of IOPS performed by the benchmark. If perf spits
> out events/sec it's kinda hard to get this.
>
That's 'perf stat -a sleep 2'
--
Do not meddle in the internals of kernels, for they are subtle and quick to panic.
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: Wiki docs on counting and tracing KVM perf events
2010-05-20 8:19 ` Avi Kivity
@ 2010-05-20 8:24 ` Jes Sorensen
2010-05-20 11:05 ` Stefan Hajnoczi
0 siblings, 1 reply; 12+ messages in thread
From: Jes Sorensen @ 2010-05-20 8:24 UTC (permalink / raw
To: Avi Kivity; +Cc: Stefan Hajnoczi, kvm
On 05/20/10 10:19, Avi Kivity wrote:
> On 05/20/2010 11:15 AM, Jes Sorensen wrote:
>>> Two things are missing to make this really useful:
>>>
>>> - a continuously updating difference mode like kvm_stat
>>> - subevents; for example kvm:kvm_exit is an aggregate of all exit types
>>> that can be split using filters to show individual exit reason
>>> statistics
>>>
>> Third missing item, which I find really useful:
>> - run once spit out raw counters
>>
>> For some operations, like file system benchmarking, it is useful to
>> sample the counters before and after and then divide the raw number of
>> events by the number of IOPS performed by the benchmark. If perf spits
>> out events/sec it's kinda hard to get this.
>>
>
> That's 'perf stat -a sleep 2'
sleep 2 doesn't really cut it, I guess you could do it with
perf stat -a sleep 0.1
but to be honest, that is pretty ugly. Something like this would be a
lot nicer:
perf stat -a -raw
Jes
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: Wiki docs on counting and tracing KVM perf events
2010-05-20 8:24 ` Jes Sorensen
@ 2010-05-20 11:05 ` Stefan Hajnoczi
2010-05-20 11:10 ` Avi Kivity
0 siblings, 1 reply; 12+ messages in thread
From: Stefan Hajnoczi @ 2010-05-20 11:05 UTC (permalink / raw
To: Jes Sorensen; +Cc: Avi Kivity, kvm
8330 kvm:kvm_entry # 0.000 M/sec
^--- count since starting perf
The 8330 number means that kvm_entry has fired 8330 times since perf
was started. Like Avi says, you need to keep the perf process
running. I run benchmarks using a script that kills perf after the
benchmark completes.
Jes, you're right, something like "perf stat -e kvm:* --start" and
"perf stat --stop" would be more usable for system-wide monitoring. I
wonder if it is possible to support this or whether the perf process
needs to periodically accumulate the counters (i.e. babysit the kernel
infrastructure)?
Stefan
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: Wiki docs on counting and tracing KVM perf events
2010-05-20 11:05 ` Stefan Hajnoczi
@ 2010-05-20 11:10 ` Avi Kivity
2010-05-20 11:16 ` Jes Sorensen
0 siblings, 1 reply; 12+ messages in thread
From: Avi Kivity @ 2010-05-20 11:10 UTC (permalink / raw
To: Stefan Hajnoczi; +Cc: Jes Sorensen, kvm
On 05/20/2010 02:05 PM, Stefan Hajnoczi wrote:
> 8330 kvm:kvm_entry # 0.000 M/sec
> ^--- count since starting perf
>
> The 8330 number means that kvm_entry has fired 8330 times since perf
> was started. Like Avi says, you need to keep the perf process
> running. I run benchmarks using a script that kills perf after the
> benchmark completes.
>
> Jes, you're right, something like "perf stat -e kvm:* --start" and
> "perf stat --stop" would be more usable for system-wide monitoring. I
> wonder if it is possible to support this or whether the perf process
> needs to periodically accumulate the counters (i.e. babysit the kernel
> infrastructure)?
>
perf needs to be running to pull data out of the kernel (and since
profiling is tied to an fd life cycle).
What's wrong with starting perf after the warm-up period and stopping it
before it's done?
--
Do not meddle in the internals of kernels, for they are subtle and quick to panic.
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: Wiki docs on counting and tracing KVM perf events
2010-05-20 11:10 ` Avi Kivity
@ 2010-05-20 11:16 ` Jes Sorensen
2010-05-20 11:23 ` Stefan Hajnoczi
0 siblings, 1 reply; 12+ messages in thread
From: Jes Sorensen @ 2010-05-20 11:16 UTC (permalink / raw
To: Avi Kivity; +Cc: Stefan Hajnoczi, kvm
On 05/20/10 13:10, Avi Kivity wrote:
> On 05/20/2010 02:05 PM, Stefan Hajnoczi wrote:
>> Jes, you're right, something like "perf stat -e kvm:* --start" and
>> "perf stat --stop" would be more usable for system-wide monitoring. I
>> wonder if it is possible to support this or whether the perf process
>> needs to periodically accumulate the counters (i.e. babysit the kernel
>> infrastructure)?
>>
>
> perf needs to be running to pull data out of the kernel (and since
> profiling is tied to an fd life cycle).
>
> What's wrong with starting perf after the warm-up period and stopping it
> before it's done?
It's pretty hard to script.
I have scripts that do 'read_stats() ; run_test() ; read_stats() ;
calc_average()'
Autotest would like to do this kinda stuff too.
Jes
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: Wiki docs on counting and tracing KVM perf events
2010-05-20 11:16 ` Jes Sorensen
@ 2010-05-20 11:23 ` Stefan Hajnoczi
2010-05-20 12:14 ` Avi Kivity
0 siblings, 1 reply; 12+ messages in thread
From: Stefan Hajnoczi @ 2010-05-20 11:23 UTC (permalink / raw
To: Jes Sorensen; +Cc: Avi Kivity, kvm
On Thu, May 20, 2010 at 12:16 PM, Jes Sorensen <Jes.Sorensen@redhat.com> wrote:
> On 05/20/10 13:10, Avi Kivity wrote:
>> What's wrong with starting perf after the warm-up period and stopping it
>> before it's done?
>
> It's pretty hard to script.
I use the following. It ain't pretty:
#!/bin/bash
cleanup() {
trap - 2
kill -2 $sleep_pid
echo 0 >/sys/kernel/debug/tracing/events/kvm/enable
kill $cat_pid
}
perf stat -a -e 'kvm:*' sleep 1h >results/perf_stat 2>&1 &
sleep_pid=$(sleep 1 && pgrep -x -f "sleep 1h") # sleep 1 is to avoid
race with forked perf process
trap cleanup 2
echo 1 >/sys/kernel/debug/tracing/events/kvm/enable
cat /sys/kernel/debug/tracing/trace_pipe >results/trace &
cat_pid=$!
# ...do stuff here...
cleanup
Stefan
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: Wiki docs on counting and tracing KVM perf events
2010-05-20 11:23 ` Stefan Hajnoczi
@ 2010-05-20 12:14 ` Avi Kivity
2010-05-20 12:24 ` Stefan Hajnoczi
0 siblings, 1 reply; 12+ messages in thread
From: Avi Kivity @ 2010-05-20 12:14 UTC (permalink / raw
To: Stefan Hajnoczi; +Cc: Jes Sorensen, kvm
On 05/20/2010 02:23 PM, Stefan Hajnoczi wrote:
> On Thu, May 20, 2010 at 12:16 PM, Jes Sorensen<Jes.Sorensen@redhat.com> wrote:
>
>> On 05/20/10 13:10, Avi Kivity wrote:
>>
>>> What's wrong with starting perf after the warm-up period and stopping it
>>> before it's done?
>>>
>> It's pretty hard to script.
>>
> I use the following. It ain't pretty:
>
> #!/bin/bash
> cleanup() {
> trap - 2
> kill -2 $sleep_pid
> echo 0>/sys/kernel/debug/tracing/events/kvm/enable
> kill $cat_pid
> }
>
> perf stat -a -e 'kvm:*' sleep 1h>results/perf_stat 2>&1&
> sleep_pid=$(sleep 1&& pgrep -x -f "sleep 1h") # sleep 1 is to avoid
> race with forked perf process
> trap cleanup 2
> echo 1>/sys/kernel/debug/tracing/events/kvm/enable
> cat /sys/kernel/debug/tracing/trace_pipe>results/trace&
> cat_pid=$!
>
> # ...do stuff here...
>
> cleanup
>
perf will enable the events by itself (no?), so all you need is is the
perf call in the middle.
What's missing is vmstat-like or kvm_stat-like output, but that's
another thing.
--
Do not meddle in the internals of kernels, for they are subtle and quick to panic.
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: Wiki docs on counting and tracing KVM perf events
2010-05-20 12:14 ` Avi Kivity
@ 2010-05-20 12:24 ` Stefan Hajnoczi
2010-05-20 12:37 ` Avi Kivity
0 siblings, 1 reply; 12+ messages in thread
From: Stefan Hajnoczi @ 2010-05-20 12:24 UTC (permalink / raw
To: Avi Kivity; +Cc: Jes Sorensen, kvm
On Thu, May 20, 2010 at 1:14 PM, Avi Kivity <avi@redhat.com> wrote:
>> echo 1>/sys/kernel/debug/tracing/events/kvm/enable
>> cat /sys/kernel/debug/tracing/trace_pipe>results/trace&
>
> perf will enable the events by itself (no?), so all you need is is the perf
> call in the middle.
Yes, it will enable events. However, I am also generating a
kvm_trace-like log using trace_pipe. On this box I couldn't get perf
trace working so I used trace_pipe for the kvm_trace-equivalent and
perf stat for the kvm_stat equivalent.
Stefan
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: Wiki docs on counting and tracing KVM perf events
2010-05-20 12:24 ` Stefan Hajnoczi
@ 2010-05-20 12:37 ` Avi Kivity
0 siblings, 0 replies; 12+ messages in thread
From: Avi Kivity @ 2010-05-20 12:37 UTC (permalink / raw
To: Stefan Hajnoczi; +Cc: Jes Sorensen, kvm
On 05/20/2010 03:24 PM, Stefan Hajnoczi wrote:
> On Thu, May 20, 2010 at 1:14 PM, Avi Kivity<avi@redhat.com> wrote:
>
>>> echo 1>/sys/kernel/debug/tracing/events/kvm/enable
>>> cat /sys/kernel/debug/tracing/trace_pipe>results/trace&
>>>
>> perf will enable the events by itself (no?), so all you need is is the perf
>> call in the middle.
>>
> Yes, it will enable events. However, I am also generating a
> kvm_trace-like log using trace_pipe. On this box I couldn't get perf
> trace working so I used trace_pipe for the kvm_trace-equivalent and
> perf stat for the kvm_stat equivalent.
>
Ah. There's the new shiny trace_cmd as well.
If you're getting a (binary) trace, runing perf stat is redundant since
it contains the same data.
--
Do not meddle in the internals of kernels, for they are subtle and quick to panic.
^ permalink raw reply [flat|nested] 12+ messages in thread
end of thread, other threads:[~2010-05-20 12:37 UTC | newest]
Thread overview: 12+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2010-05-13 14:35 Wiki docs on counting and tracing KVM perf events Stefan Hajnoczi
2010-05-13 14:57 ` Avi Kivity
2010-05-20 8:15 ` Jes Sorensen
2010-05-20 8:19 ` Avi Kivity
2010-05-20 8:24 ` Jes Sorensen
2010-05-20 11:05 ` Stefan Hajnoczi
2010-05-20 11:10 ` Avi Kivity
2010-05-20 11:16 ` Jes Sorensen
2010-05-20 11:23 ` Stefan Hajnoczi
2010-05-20 12:14 ` Avi Kivity
2010-05-20 12:24 ` Stefan Hajnoczi
2010-05-20 12:37 ` Avi Kivity
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.