From: John Snow <jsnow@redhat.com>
To: Stefan Hajnoczi <stefanha@redhat.com>,
"Niteesh G. S." <niteesh.gs@gmail.com>
Cc: kwolf@redhat.com, wainersm@redhat.com, qemu-devel@nongnu.org,
ehabkost@redhat.com, armbru@redhat.com
Subject: Re: GSoC Intro - TUI interface for QMP
Date: Mon, 24 May 2021 13:34:19 -0400 [thread overview]
Message-ID: <351d1344-a243-ca77-396b-0c1d071b9066@redhat.com> (raw)
In-Reply-To: <YKuq242kdKxhvHAr@stefanha-x1.localdomain>
On 5/24/21 9:32 AM, Stefan Hajnoczi wrote:
> On Sat, May 22, 2021 at 12:32:00AM +0530, Niteesh G. S. wrote:
Welcome Niteesh :) I look forward to working with you this summer.
>> By end of this summer, I would like to get a basic TUI with some desirable
>> features working. Some of the features I would like to get working are
As a reminder to anyone reading this thread, the goal is to create a
qmp-shell that functions more as a TUI, akin to mutt, irssi, or (my
favorite example) mitmproxy. The idea is that there will be, at minimum,
a history panel showing QMP messages that have occurred so far and a
text entry panel for entering new commands.
This shell can then be augmented with various other features to
facilitate testing, debugging, etc. One of the core upgrades over the
existing qmp-shell will be the featuring of truly asynchronous events
which will appear in the history panel without requiring the human user
to press <enter> to allow them to display. This will use a new
Asynchronous QMP library to facilitate this feature, bringing with it
fixes over our current use of undocumented Python features abusing
non-blocking sockets in the old QMP library.
My plan is to worry about implementing the very basics of the shell
first, and then to add more features on as we feel comfortable with the
basics. We can discuss what we consider to be the bare minimum for this
project and lay out the feature requirements that define a successful
minimum.
>> 1) Syntax checking
To a limited extent. I don't want to disallow the user from sending
commands that are invalid in the event that we want to test the server's
ability to cope with and reply to invalid commands.
However, if the syntax is malformed enough that we can't understand it
to send it to the server, good error messages that point out what
exactly went wrong are helpful.
>> 2) Syntax highlighting
>> 3) Code completion
>> 4) Logging
>>
>> I would like to hear some of the features you would like to have and also
>> your
>> advice's on implementation.
>
> Welcome Niteesh!
>
> It would be great to have an QMP API documentation viewer built into the
> TUI. When you enter a command like 'migrate' it would automatically
> display the associated QMP command documentation from qapi/*.json.
>
Stefan's suggestion is an important one to me. The current qmp-shell
does have an auto-complete list of available commands (press <tab> after
connecting to see the list), but it offers no explanation of what the
commands do, what their arguments are, etc.
Being able to get interactive help explaining the commands, their
arguments, etc would be a massive usability improvement over the old shell.
> Stefan
>
Additionally, there's a few threads we've had before with people's
stretch goals, dreams, nice-to-haves and so on. I owe you a bit of a
compilation of those ideas. Here are some that I happen to remember.
(Keep in mind that not all of these have been discussed or vetted
thoroughly on list, some may be vastly harder than others, and not all
of them may be mutually compatible. There may even be objections to some
of these. We will definitely not do every last thing on this list, and
it is important we stay focused on the core task and get code merged
this summer. However, some of these stretch goals might be fun and
provide you with extreme leeway to investigate features that interest
you personally, so I am listing them now.)
1. Utilize or otherwise re-implement readline's history feature, such
that when opening qmp-shell, you can press up/down to recall previously
issued commands. (The usage of 'readline' library itself may be
complicated if we wish to use a TUI library like urwid. Some re-creation
of features may be necessary.)
2. Implement tab-complete for command names, command arguments, etc.
3. Implement a FiSH-like suggestion mode where, without tab complete,
the text entry window shows (in e.g. a gray font) the most likely
auto-complete. You can press <tab> to accept this auto-completion.
4. Add the ability to save a log file for a given session, recording
timestamps, incoming messages and outgoing messages. Sessions could
possibly be automatically saved and cycled such that the last 10 or so
are kept, or maybe up 10MB, or some other kind of limit.
5. Add the ability to load an old log file and "replay" it to a server.
Optionally, the "replay" might be able to use the log timestamps to
replay the commands with an identical timing. After a replay, it might
be nice to be able to use a diff viewer of some kind to show differences
between the original script and the replay.
6. Add the ability to "attach" qmp-shell to an existing instance of a
libvirt-managed QEMU and have qmp-shell behave in a "watch" mode,
showing commands sent back and forth between QEMU and libvirt, to help
facilitate live debugging of libvirt.
7. Add the ability to "hide" certain events from the history view. By
pressing a certain hotkey, we can enter a command mode and start
filtering certain events. For example, we may wish to hide from view all
JOB_STATUS_CHANGE events from view. Certain filtering modes may warrant
their own explicit mode available directly in the shell, but we may wish
to provide a more powerful filtering/scripting mechanism.
8. Similar to above, add the ability to change the "verbosity" of events
in the history panel live at runtime. For example:
- Show the full QMP message, pretty-printed
- Show the full QMP message, condensed with no spaces.
- Show the raw byte sequence of each QMP message.
- Show an abbreviated summary; just the command name or event name.
- Show or hide timestamps
- Show outgoing commands in terms of what the user typed in qmp-shell.
(e.g., if we offer a simple syntax, show the simple-syntax version.)
Being able to toggle between these at-will may be nice to help see
more or less information dynamically as the situation calls for it.
9. The ability to generate an iotest stub from the current session
history or a subsection thereof. It's OK if the test isn't fully
functional on its own, leaving some sections as stubs. Generating the
commands, waiting for the commands to complete, and generating an '.out'
could be a nice feature that helps qmp-shell users quickly write tests.
Some thoughts on this:
- iotests usually launch a VM to connect to. qmp-shell will not know the
arguments QEMU was launched with. The generated test might attempt to
launch QEMU with a minimum viable configuration, leaving other arguments
as a stub marked #FIXME.
- The generated iotest may be able to automatically generate event
waits. For instance, if a JOB_STATUS_CHANGE event occurs prior to the
user entering the next command, the generated iotest could automatically
insert a delay that waits for the same event.
- Generated tests should always end with a VM shutdown sequence, either
explicitly (user sends a 'quit' command) or implicitly (The python
iotest library calls vm.shutdown())
10. The ability to detect a disconnection and attempt to reconnect
automatically, similar to how an IRC client would.
11. The ability to define macros to perform multi-step, complex routines
with a single command. Something like a ~/.config/qemu/qmp-shell.conf
file could possibly define these macros. The ability to create/save/edit
these macros directly from the shell could be nice.
12. Integrating tools like Vladimir's block graph visualization tool.
There are complex memory structures in QEMU that can be hard to
understand quickly by reading large volumes of text. qmp-shell could be
augmented to help render and visualize them quickly, facilitating
debugging greatly.
I'll update this thread with more ideas as I remember them when going
back through previous upstream discussions we've had on the topic. We
may want to create a wiki page https://wiki.qemu.org/Documentation and
start updating it with various ideas, stretch goals, and so on, to keep
our planning organized. If you do not have an account on this wiki yet,
I believe you'll need to ask the list for an account, letting us know
what email and username you want to use for it.
...I also need to warn you about one big problem that looms over this
project: QMP is a syntax designed for machines, but it's difficult to
write by hand. The old qmp-shell has a simplistic syntax that was
designed to make entering commands easier, but it isn't a suitable
syntax for entering more complex, hierarchical data.
One of the decisions we'll need to make is if we simply REMOVE the
convenient syntax (allowing only the raw input of QMP), or if we attempt
to design a new shorthand syntax, simply use the old shorthand syntax, etc.
This is a complex area (and one I haven't fully solved for myself), so
if this project involves only the input of *raw QMP commands* into the
shell, I would still consider that a success if we got some of the other
features worked into it. We will have to discuss the pros/cons of adding
an imperfect shorthand syntax to this shell; there has been some
reluctance to that idea in the past.
So, your first homework assignment from me is:
- Create a QEMU wiki account
- Port your GSoC proposal to the wiki
- Begin recording stretch goals to that wiki page
- Set up an (at least) weekly meeting time with me to discuss progress,
problems, etc. We can chat on IRC, via google meet/zoom, etc at your
preference.
- Make sure you have a gitlab account. I will add you as a contributor
to my fork of QEMU and we can use the issue and milestone tracker to
help set expectations and stay focused.
Thanks! (And welcome!)
--js
next prev parent reply other threads:[~2021-05-24 17:37 UTC|newest]
Thread overview: 15+ messages / expand[flat|nested] mbox.gz Atom feed top
[not found] <CAN6ztm-J2GoQKkLb=Az0H2Q8UKK4oE3PgXg7g14=T53sQAUyDg@mail.gmail.com>
2021-05-21 19:02 ` Fwd: GSoC Intro - TUI interface for QMP Niteesh G. S.
2021-05-24 13:32 ` Stefan Hajnoczi
2021-05-24 17:34 ` John Snow [this message]
2021-06-02 11:08 ` Bruno Piazera Larsen
2021-06-11 14:06 ` Vladimir Sementsov-Ogievskiy via
2021-05-26 15:35 ` Fwd: " Niteesh G. S.
2021-06-01 23:47 ` John Snow
2021-06-08 15:01 ` Markus Armbruster
2021-06-08 15:49 ` John Snow
2021-06-09 11:56 ` Markus Armbruster
2021-06-09 17:06 ` John Snow
2021-06-10 7:19 ` Markus Armbruster
2021-06-09 12:07 ` Daniel P. Berrangé
2021-06-10 14:35 ` John Snow
2021-06-09 12:03 ` Daniel P. Berrangé
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=351d1344-a243-ca77-396b-0c1d071b9066@redhat.com \
--to=jsnow@redhat.com \
--cc=armbru@redhat.com \
--cc=ehabkost@redhat.com \
--cc=kwolf@redhat.com \
--cc=niteesh.gs@gmail.com \
--cc=qemu-devel@nongnu.org \
--cc=stefanha@redhat.com \
--cc=wainersm@redhat.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).