From: "Simmons, James via lustre-devel" <lustre-devel@lists.lustre.org>
To: "Bertschinger, Thomas Andrew Hjorth" <bertschinger@lanl.gov>,
"lustre-devel@lists.lustre.org" <lustre-devel@lists.lustre.org>
Subject: Re: [lustre-devel] [EXTERNAL] Re: lctl debug_kernel: usability improvements?
Date: Sun, 9 Jul 2023 22:27:32 +0000 [thread overview]
Message-ID: <51201441-1C7D-4D6B-A5F3-1D76439BD90C@ornl.gov> (raw)
[-- Attachment #1.1: Type: text/plain, Size: 4085 bytes --]
>After looking into this more I see there is a fair challenge associated with this idea. With each CPU (and execution context) having its own distinct buffer for messages (struct cfs_trace_cpu_data), the >messages are not chronologically sorted in kernel memory. Instead, they are written to a regular file in CPU order and then the sorted chronologically in userspace prior to printing.
>
>Implementing a "/dev/lmsg" device would be challenging with the existing data structures because sorting would have to happen in kernel space as the device responds to reads.
>
>I found this issue: LU-14428 "Convert tracefile to use ring_buffer from linux" which does not look to be completed, seeing as a ring_buffer is not currently in use here -- but if this is completed, >implementing "/dev/lmsg" with the same interface as "/dev/kmsg" would be much simpler. (Assuming I understand correctly that the proposal is a single global ring buffer. Let me know if I am mistaken >and the proposal is a set of per-CPU ring buffers, because then the sorting problem is not avoided.)
>
>I reported a new issue LU-16746 "Convert tracefile to export debug logs via character device" for this idea.This can be worked on after LU-14428 is completed. If I can be of assistance on LU-14428 by >helping with any sub-tasks, let me know. I am interested in helping with this area of Lustre.
Hello.
Sorry about the delay. Been busy with the IPv6 work. I do want to let you know Neil Brown resolved the issue with the last LU-14428 patch. Its got the reviews to land to the master branch. I CC Neil Brown for feed back about your ideas.
________________________________
From: lustre-devel <lustre-devel-bounces@lists.lustre.org> on behalf of Bertschinger, Thomas Andrew Hjorth via lustre-devel <lustre-devel@lists.lustre.org>
Sent: Monday, April 10, 2023 8:19 AM
To: lustre-devel@lists.lustre.org
Subject: [lustre-devel] lctl debug_kernel: usability improvements?
Hello,
Recently when using "lctl dk" I have found myself wanting some of the "quality of life" features that exist in the similar linux tool dmesg. In particular, having the ability to "follow" the debug log like "dmesg -w" would be very handy IMO.
I've attempted to implement this in userspace with the existing tooling (using "lctl debug_daemon" to write the encoded log to a file, and "lctl debug_file" to decode it) but have run into challenges. I first tried creating a FIFO and had debug_daemon write to it and debug_file read from it. Unfortunately this fails because the kernel thread that writes to this file (tracefiled in libcfs/libcfs/tracefile.c) repeatedly opens and closes the file, but after the first close reading the FIFO fails.
My next idea was to have debug_daemon write to a regular file and debug_file read it like "tail -f". This should work in theory but has disadvantages: the user must remember to delete the file when done (the tool could do this but not if it exits uncleanly), and also entries could be missed if the file is deleted while the tool is still running.
I think the cleanest solution is to rework the debug_kernel interface to be like linux's /dev/kmsg. A character device, perhaps named /dev/lmsg, could be created that outputs the buffer contents when read. Implementing "follow" would be trivial with this interface. The existing userspace tools could also easily be updated to use this interface, and it would bring other benefits, for example "lctl dk" not needing to copy the message buffer to a tmp file. The disadvantage here is that this could be a significant kernel-side refactor.
I feel the ability to follow Lustre's debug log would be useful to both sysadmins and developers but want to get some other input. Would this be valuable to anyone? If this would be useful -- and feasible -- I would be happy to submit a JIRA ticket and work on a patch but wanted to get some more opinions. I'm not very familiar with the kernel side code yet so I'm not sure how complicated this would be.
- Thomas Bertschinger
________________________________
[-- Attachment #1.2: Type: text/html, Size: 8031 bytes --]
[-- Attachment #2: Type: text/plain, Size: 165 bytes --]
_______________________________________________
lustre-devel mailing list
lustre-devel@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-devel-lustre.org
reply other threads:[~2023-07-09 22:27 UTC|newest]
Thread overview: [no followups] expand[flat|nested] mbox.gz Atom feed
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=51201441-1C7D-4D6B-A5F3-1D76439BD90C@ornl.gov \
--to=lustre-devel@lists.lustre.org \
--cc=bertschinger@lanl.gov \
--cc=simmonsja@ornl.gov \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).