From: "Darrick J. Wong" <djwong@kernel.org>
To: Kent Overstreet <kent.overstreet@linux.dev>
Cc: akpm@linux-foundation.org, daniel@gluo.nz,
linux-xfs@vger.kernel.org, linux-bcachefs@vger.kernel.org,
linux-kernel@vger.kernel.org
Subject: Re: [PATCH 09/10] time_stats: report information in json format
Date: Fri, 23 Feb 2024 22:02:50 -0800 [thread overview]
Message-ID: <20240224060250.GQ6226@frogsfrogsfrogs> (raw)
In-Reply-To: <hf4u56xx3riqz2wyx3qxqiidccocu6cs5z5qdla3zgo5v3wcbl@dldlaaamx2kn>
On Sat, Feb 24, 2024 at 12:10:33AM -0500, Kent Overstreet wrote:
> On Fri, Feb 23, 2024 at 08:15:45PM -0800, Darrick J. Wong wrote:
> > On Fri, Feb 23, 2024 at 05:12:26PM -0800, Darrick J. Wong wrote:
> > > From: Darrick J. Wong <djwong@kernel.org>
> > >
> > > Export json versions of time statistics information. Given the tabular
> > > nature of the numbers exposed, this will make it a lot easier for higher
> > > (than C) level languages (e.g. python) to import information without
> > > needing to write yet another clumsy string parser.
> > >
> > > Signed-off-by: Darrick J. Wong <djwong@kernel.org>
> > > Signed-off-by: Kent Overstreet <kent.overstreet@linux.dev>
> > > ---
> > > include/linux/time_stats.h | 2 +
> > > lib/time_stats.c | 87 ++++++++++++++++++++++++++++++++++++++++++++
> > > 2 files changed, 89 insertions(+)
> > >
> > >
> > > diff --git a/include/linux/time_stats.h b/include/linux/time_stats.h
> > > index b3c810fff963a..4e1f5485ed039 100644
> > > --- a/include/linux/time_stats.h
> > > +++ b/include/linux/time_stats.h
> > > @@ -156,6 +156,8 @@ static inline bool track_event_change(struct time_stats *stats, bool v)
> > > struct seq_buf;
> > > void time_stats_to_seq_buf(struct seq_buf *, struct time_stats *,
> > > const char *epoch_name, unsigned int flags);
> > > +void time_stats_to_json(struct seq_buf *, struct time_stats *,
> > > + const char *epoch_name, unsigned int flags);
> > >
> > > void time_stats_exit(struct time_stats *);
> > > void time_stats_init(struct time_stats *);
> > > diff --git a/lib/time_stats.c b/lib/time_stats.c
> > > index 0fb3d854e503b..c0f209dd9f6dd 100644
> > > --- a/lib/time_stats.c
> > > +++ b/lib/time_stats.c
> > > @@ -266,6 +266,93 @@ void time_stats_to_seq_buf(struct seq_buf *out, struct time_stats *stats,
> > > }
> > > EXPORT_SYMBOL_GPL(time_stats_to_seq_buf);
> > >
> > > +void time_stats_to_json(struct seq_buf *out, struct time_stats *stats,
> > > + const char *epoch_name, unsigned int flags)
> > > +{
> > > + struct quantiles *quantiles = time_stats_to_quantiles(stats);
> > > + s64 f_mean = 0, d_mean = 0;
> > > + u64 f_stddev = 0, d_stddev = 0;
> > > +
> > > + if (stats->buffer) {
> > > + int cpu;
> > > +
> > > + spin_lock_irq(&stats->lock);
> > > + for_each_possible_cpu(cpu)
> > > + __time_stats_clear_buffer(stats, per_cpu_ptr(stats->buffer, cpu));
> > > + spin_unlock_irq(&stats->lock);
> > > + }
> > > +
> > > + if (stats->freq_stats.n) {
> > > + /* avoid divide by zero */
> > > + f_mean = mean_and_variance_get_mean(stats->freq_stats);
> > > + f_stddev = mean_and_variance_get_stddev(stats->freq_stats);
> > > + d_mean = mean_and_variance_get_mean(stats->duration_stats);
> > > + d_stddev = mean_and_variance_get_stddev(stats->duration_stats);
> > > + } else if (flags & TIME_STATS_PRINT_NO_ZEROES) {
> > > + /* unless we didn't want zeroes anyway */
> > > + return;
> > > + }
> > > +
> > > + seq_buf_printf(out, "{\n");
> > > + seq_buf_printf(out, " \"epoch\": \"%s\",\n", epoch_name);
> > > + seq_buf_printf(out, " \"count\": %llu,\n", stats->duration_stats.n);
> > > +
> > > + seq_buf_printf(out, " \"duration_ns\": {\n");
> > > + seq_buf_printf(out, " \"min\": %llu,\n", stats->min_duration);
> > > + seq_buf_printf(out, " \"max\": %llu,\n", stats->max_duration);
> > > + seq_buf_printf(out, " \"total\": %llu,\n", stats->total_duration);
> > > + seq_buf_printf(out, " \"mean\": %llu,\n", d_mean);
> > > + seq_buf_printf(out, " \"stddev\": %llu\n", d_stddev);
> > > + seq_buf_printf(out, " },\n");
> > > +
> > > + d_mean = mean_and_variance_weighted_get_mean(stats->duration_stats_weighted, TIME_STATS_MV_WEIGHT);
> > > + d_stddev = mean_and_variance_weighted_get_stddev(stats->duration_stats_weighted, TIME_STATS_MV_WEIGHT);
> > > +
> > > + seq_buf_printf(out, " \"duration_ewma_ns\": {\n");
> > > + seq_buf_printf(out, " \"mean\": %llu,\n", d_mean);
> > > + seq_buf_printf(out, " \"stddev\": %llu\n", d_stddev);
> > > + seq_buf_printf(out, " },\n");
> > > +
> > > + seq_buf_printf(out, " \"frequency_ns\": {\n");
> >
> > I took the variable names too literally here; these labels really ought
> > to be "between_ns" and "between_ewma_ns" to maintain consistency with
> > the labels in the table format.
> >
> > > + seq_buf_printf(out, " \"min\": %llu,\n", stats->min_freq);
> > > + seq_buf_printf(out, " \"max\": %llu,\n", stats->max_freq);
> > > + seq_buf_printf(out, " \"mean\": %llu,\n", f_mean);
> > > + seq_buf_printf(out, " \"stddev\": %llu\n", f_stddev);
> > > + seq_buf_printf(out, " },\n");
> > > +
> > > + f_mean = mean_and_variance_weighted_get_mean(stats->freq_stats_weighted, TIME_STATS_MV_WEIGHT);
> > > + f_stddev = mean_and_variance_weighted_get_stddev(stats->freq_stats_weighted, TIME_STATS_MV_WEIGHT);
> > > +
> > > + seq_buf_printf(out, " \"frequency_ewma_ns\": {\n");
> > > + seq_buf_printf(out, " \"mean\": %llu,\n", f_mean);
> > > + seq_buf_printf(out, " \"stddev\": %llu\n", f_stddev);
> > > +
> > > + if (quantiles) {
> > > + u64 last_q = 0;
> > > +
> > > + /* close frequency_ewma_ns but signal more items */
> >
> > (also this comment)
> >
> > > + seq_buf_printf(out, " },\n");
> > > +
> > > + seq_buf_printf(out, " \"quantiles_ns\": [\n");
> > > + eytzinger0_for_each(i, NR_QUANTILES) {
> > > + bool is_last = eytzinger0_next(i, NR_QUANTILES) == -1;
> > > +
> > > + u64 q = max(quantiles->entries[i].m, last_q);
> > > + seq_buf_printf(out, " %llu", q);
> > > + if (!is_last)
> > > + seq_buf_printf(out, ", ");
> > > + last_q = q;
> > > + }
> > > + seq_buf_printf(out, " ]\n");
> > > + } else {
> > > + /* close frequency_ewma_ns without dumping further */
> >
> > (this one too)
> >
> > Kent, would you mind making that edit the next time you reflow your
> > branch?
> >
> > --D
> >
> > > + seq_buf_printf(out, " }\n");
> > > + }
> > > +
> > > + seq_buf_printf(out, "}\n");
> > > +}
> > > +EXPORT_SYMBOL_GPL(time_stats_to_json);
> > > +
> > > void time_stats_exit(struct time_stats *stats)
> > > {
> > > free_percpu(stats->buffer);
> > >
> > >
>
>
> From 5885a65fa5a0aace7bdf1a8fa58ac2bca3b15900 Mon Sep 17 00:00:00 2001
> From: Kent Overstreet <kent.overstreet@linux.dev>
> Date: Sat, 24 Feb 2024 00:10:06 -0500
> Subject: [PATCH] fixup! time_stats: report information in json format
>
>
> diff --git a/lib/time_stats.c b/lib/time_stats.c
> index 0b90c80cba9f..d7dd64baebb8 100644
> --- a/lib/time_stats.c
> +++ b/lib/time_stats.c
> @@ -313,7 +313,7 @@ void time_stats_to_json(struct seq_buf *out, struct time_stats *stats,
> seq_buf_printf(out, " \"stddev\": %llu\n", d_stddev);
> seq_buf_printf(out, " },\n");
>
> - seq_buf_printf(out, " \"frequency_ns\": {\n");
> + seq_buf_printf(out, " \"between_ns\": {\n");
> seq_buf_printf(out, " \"min\": %llu,\n", stats->min_freq);
> seq_buf_printf(out, " \"max\": %llu,\n", stats->max_freq);
> seq_buf_printf(out, " \"mean\": %llu,\n", f_mean);
> @@ -323,14 +323,14 @@ void time_stats_to_json(struct seq_buf *out, struct time_stats *stats,
> f_mean = mean_and_variance_weighted_get_mean(stats->freq_stats_weighted, TIME_STATS_MV_WEIGHT);
> f_stddev = mean_and_variance_weighted_get_stddev(stats->freq_stats_weighted, TIME_STATS_MV_WEIGHT);
>
> - seq_buf_printf(out, " \"frequency_ewma_ns\": {\n");
> + seq_buf_printf(out, " \"between_ewma_ns\": {\n");
Looks good to me,
Reviewed-by: Darrick J. Wong <djwong@kernel.org>
--D
> seq_buf_printf(out, " \"mean\": %llu,\n", f_mean);
> seq_buf_printf(out, " \"stddev\": %llu\n", f_stddev);
>
> if (quantiles) {
> u64 last_q = 0;
>
> - /* close frequency_ewma_ns but signal more items */
> + /* close between_ewma_ns but signal more items */
> seq_buf_printf(out, " },\n");
>
> seq_buf_printf(out, " \"quantiles_ns\": [\n");
> @@ -345,7 +345,7 @@ void time_stats_to_json(struct seq_buf *out, struct time_stats *stats,
> }
> seq_buf_printf(out, " ]\n");
> } else {
> - /* close frequency_ewma_ns without dumping further */
> + /* close between_ewma_ns without dumping further */
> seq_buf_printf(out, " }\n");
> }
>
next prev parent reply other threads:[~2024-02-24 6:02 UTC|newest]
Thread overview: 51+ messages / expand[flat|nested] mbox.gz Atom feed top
2024-02-24 1:00 [PATCHBOMB] time_stats, thread_with_file: lifting generic code to lib Darrick J. Wong
2024-02-24 1:07 ` [PATCHSET 1/6] time_stats: promote to lib/ Darrick J. Wong
2024-02-24 1:09 ` [PATCH 1/4] mean and variance: Promote to lib/math Darrick J. Wong
2024-02-24 1:09 ` [PATCH 2/4] eytzinger: Promote to include/linux/ Darrick J. Wong
2024-02-24 1:09 ` [PATCH 3/4] bcachefs: bch2_time_stats_to_seq_buf() Darrick J. Wong
2024-02-24 1:10 ` [PATCH 4/4] time_stats: Promote to lib/ Darrick J. Wong
2024-02-24 1:08 ` [PATCHSET 2/6] time_stats: cleanups and fixes Darrick J. Wong
2024-02-24 1:10 ` [PATCH 01/10] time_stats: report lifetime of the stats object Darrick J. Wong
2024-02-24 1:10 ` [PATCH 02/10] time_stats: split stats-with-quantiles into a separate structure Darrick J. Wong
2024-02-24 1:10 ` [PATCH 03/10] time_stats: fix struct layout bloat Darrick J. Wong
2024-02-24 1:11 ` [PATCH 04/10] time_stats: add larger units Darrick J. Wong
2024-02-24 1:11 ` [PATCH 05/10] time_stats: don't print any output if event count is zero Darrick J. Wong
2024-02-24 1:11 ` [PATCH 06/10] time_stats: allow custom epoch names Darrick J. Wong
2024-02-24 1:11 ` [PATCH 07/10] mean_and_variance: put struct mean_and_variance_weighted on a diet Darrick J. Wong
2024-02-24 1:12 ` [PATCH 08/10] time_stats: shrink time_stat_buffer for better alignment Darrick J. Wong
2024-02-24 1:12 ` [PATCH 09/10] time_stats: report information in json format Darrick J. Wong
2024-02-24 4:15 ` Darrick J. Wong
2024-02-24 5:10 ` Kent Overstreet
2024-02-24 6:02 ` Darrick J. Wong [this message]
2024-02-24 1:12 ` [PATCH 10/10] time_stats: Kill TIME_STATS_HAVE_QUANTILES Darrick J. Wong
2024-02-24 1:08 ` [PATCHSET RFC 3/6] xfs: capture statistics about wait times Darrick J. Wong
2024-02-24 1:12 ` [PATCH 1/4] xfs: present wait time statistics Darrick J. Wong
2024-02-24 1:13 ` [PATCH 2/4] xfs: present time stats for scrubbers Darrick J. Wong
2024-02-24 1:13 ` [PATCH 3/4] xfs: present timestats in json format Darrick J. Wong
2024-02-24 1:13 ` [PATCH 4/4] xfs: create debugfs uuid aliases Darrick J. Wong
2024-02-24 1:08 ` [PATCHSET 4/6] thread_with_file: promote to lib/ Darrick J. Wong
2024-02-24 1:14 ` [PATCH 01/10] bcachefs: thread_with_stdio: eliminate double buffering Darrick J. Wong
2024-02-24 1:14 ` [PATCH 02/10] bcachefs: thread_with_stdio: convert to darray Darrick J. Wong
2024-02-24 1:14 ` [PATCH 03/10] bcachefs: thread_with_stdio: kill thread_with_stdio_done() Darrick J. Wong
2024-02-24 1:14 ` [PATCH 04/10] bcachefs: thread_with_stdio: fix bch2_stdio_redirect_readline() Darrick J. Wong
2024-02-24 1:15 ` [PATCH 05/10] bcachefs: Thread with file documentation Darrick J. Wong
2024-02-24 1:15 ` [PATCH 06/10] darray: lift from bcachefs Darrick J. Wong
2024-02-24 1:15 ` [PATCH 07/10] thread_with_file: Lift " Darrick J. Wong
2024-02-24 1:15 ` [PATCH 08/10] thread_with_stdio: Mark completed in ->release() Darrick J. Wong
2024-02-24 1:16 ` [PATCH 09/10] kernel/hung_task.c: export sysctl_hung_task_timeout_secs Darrick J. Wong
2024-02-24 1:16 ` [PATCH 10/10] thread_with_stdio: suppress hung task warning Darrick J. Wong
2024-02-24 1:08 ` [PATCHSET 5/6] thread_with_file: cleanups and fixes Darrick J. Wong
2024-02-24 1:16 ` [PATCH 1/5] thread_with_file: allow creation of readonly files Darrick J. Wong
2024-02-24 1:16 ` [PATCH 2/5] thread_with_file: fix various printf problems Darrick J. Wong
2024-02-24 1:17 ` [PATCH 3/5] thread_with_file: create ops structure for thread_with_stdio Darrick J. Wong
2024-02-24 1:17 ` [PATCH 4/5] thread_with_file: allow ioctls against these files Darrick J. Wong
2024-02-24 1:17 ` [PATCH 5/5] thread_with_file: Fix missing va_end() Darrick J. Wong
2024-02-24 1:09 ` [PATCHSET RFC 6/6] xfs: live health monitoring of filesystems Darrick J. Wong
2024-02-24 1:17 ` [PATCH 1/8] xfs: use thread_with_file to create a monitoring file Darrick J. Wong
2024-02-24 1:18 ` [PATCH 2/8] xfs: create hooks for monitoring health updates Darrick J. Wong
2024-02-24 1:18 ` [PATCH 3/8] xfs: create a filesystem shutdown hook Darrick J. Wong
2024-02-24 1:18 ` [PATCH 4/8] xfs: report shutdown events through healthmon Darrick J. Wong
2024-02-24 1:18 ` [PATCH 5/8] xfs: report metadata health " Darrick J. Wong
2024-02-24 1:19 ` [PATCH 6/8] xfs: report media errors " Darrick J. Wong
2024-02-24 1:19 ` [PATCH 7/8] xfs: allow reconfiguration of the health monitoring device Darrick J. Wong
2024-02-24 1:19 ` [PATCH 8/8] xfs: send uevents when mounting and unmounting a filesystem Darrick J. Wong
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20240224060250.GQ6226@frogsfrogsfrogs \
--to=djwong@kernel.org \
--cc=akpm@linux-foundation.org \
--cc=daniel@gluo.nz \
--cc=kent.overstreet@linux.dev \
--cc=linux-bcachefs@vger.kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-xfs@vger.kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).