From: Luis Chamberlain <mcgrof@kernel.org>
To: kdevops@lists.linux.dev
Cc: dave@stgolabs.net, a.manzanares@samsung.com, fan.ni@samsung.com,
jim.harris@samsung.com, Luis Chamberlain <mcgrof@kernel.org>
Subject: [PATCH] guestfs: add pcie passthrough XML generation / attachsupport
Date: Thu, 7 Mar 2024 18:53:38 -0800 [thread overview]
Message-ID: <20240308025338.1768534-1-mcgrof@kernel.org> (raw)
Most of the hard work to support PCIE passthrough support was already
done before for vagrant and through 'make dynconfig' for the dynamic
Kconfig stuff. This let's associate a PCIE device to a guest or lets us
pick which device onto which guest to peg a device to.
The only thing left to do for guestfs was to generate the XML files and
then attach them. When and how we do this was quite tricky though.
We just generate an XML file with the input source information, we do
this at 'make' time.
CXL folks may want to pay attention to this, it means you can use
XML output files for each target guest, or you dynamically generate
guests and their respective PCI topology at bring up time later.
To attach we just wait after we define the guest. As soon as we
define it, we can virsh attach devices to it.
To simplify things further, instead of assuming we know where exactly
on the guest we want to put a device we let libvirt figure that out.
This proves to be super clean and removes all the PCI passthrough
stuff from the XML input source file. This should also mean the CXL
dynamic topology we may want to later add support for could all be
kept out of the file source file and instead consist of XML input
files post device definition. All we'd really need for the XML device
to support CXL is to define the PCI-E reqs before we virsh define each
guest, and that is pretty generic.
Screenshot:
kdevops@gdemo ~ $ sudo nvme list
Node Generic SN Model Namespace Usage Format FW Rev
--------------------- --------------------- -------------------- ---------------------------------------- --------- -------------------------- ---------------- --------
/dev/nvme0n1 /dev/ng0n1 S332NXAH300376 SAMSUNG MZQLW1T9HMJP-000ZU 1 119.93 GB / 1.92 TB 4 KiB + 0 B CXV8301Q
Signed-off-by: Luis Chamberlain <mcgrof@kernel.org>
---
.../gen_pcie_passthrough_guestfs_xml.py | 78 +++++++++++++++++++
playbooks/roles/gen_nodes/tasks/main.yml | 13 ++++
scripts/bringup_guestfs.sh | 4 +
3 files changed, 95 insertions(+)
create mode 100755 playbooks/roles/gen_nodes/python/gen_pcie_passthrough_guestfs_xml.py
diff --git a/playbooks/roles/gen_nodes/python/gen_pcie_passthrough_guestfs_xml.py b/playbooks/roles/gen_nodes/python/gen_pcie_passthrough_guestfs_xml.py
new file mode 100755
index 000000000000..fcf96002e608
--- /dev/null
+++ b/playbooks/roles/gen_nodes/python/gen_pcie_passthrough_guestfs_xml.py
@@ -0,0 +1,78 @@
+#!/usr/bin/python3
+#
+# add_pcie_passthrough_guestfs
+#
+# For each kdevops guest, gets its PCIe passthrough devices on the yaml file
+# and augment them on the XML file.
+
+import yaml
+import json
+import sys
+import pprint
+import subprocess
+import time
+import os
+from pathlib import Path
+
+pcie_hotplug_template = """<!-- PCIE passthrough device -->
+ <hostdev mode='subsystem' type='pci' managed='yes'>
+ <driver name='vfio'/>
+ <source>
+ <address domain='0x{domain}' bus='0x{bus}' slot='0x{slot}' function='0x{function}'/>
+ </source>
+ </hostdev>
+<!-- End of PCIE passthrough device -->
+"""
+
+def main():
+ topdir = os.environ.get('TOPDIR', '.')
+
+ # load extra_vars
+ with open(f'{topdir}/extra_vars.yaml') as stream:
+ extra_vars = yaml.safe_load(stream)
+
+ yaml_nodes_file = f'{topdir}/{extra_vars["kdevops_nodes"]}'
+
+ # slurp in the guestfs_nodes list
+ with open(yaml_nodes_file) as stream:
+ nodes = yaml.safe_load(stream)
+
+ # add pcie devices
+ for node in nodes['guestfs_nodes']:
+ name = node['name']
+ pcipassthrough = node.get('pcipassthrough')
+ if not pcipassthrough:
+ continue
+ for dev_key_name in pcipassthrough:
+ dev = pcipassthrough.get(dev_key_name)
+ dev_keys = list(dev.keys())
+ if 'domain' not in dev_keys or 'bus' not in dev_keys or 'slot' not in dev_keys or 'function' not in dev_keys:
+ raise Exception(f"Missing pcie attributes for device %s in %s" %
+ (dev_key_name, yaml_nodes_file))
+ domain = dev.get('domain')
+ bus = dev.get('bus')
+ slot = dev.get('slot')
+ function = dev.get('function')
+
+ pcie_xml = f"{extra_vars['guestfs_path']}/{name}/pcie_passthrough_" + dev_key_name + ".xml"
+
+ if os.path.exists(pcie_xml):
+ os.remove(pcie_xml)
+
+ device_xml = open(pcie_xml, 'w')
+ context = {
+ "domain" : domain,
+ "bus" : bus,
+ "slot" : slot,
+ "function" : function,
+ }
+ device_xml.write(pcie_hotplug_template.format(**context))
+ device_xml.close()
+
+# result = subprocess.run(['virsh', 'attach-device', name, '--file', device_xml, '--config'], capture_output=True)
+# if result.returncode != 0:
+# raise Exception(f"Unable to vish attach-device to guest %s pci passthrough device %s using %s" %
+# (name, dev_key_name, pcie_xml))
+
+if __name__ == "__main__":
+ main()
diff --git a/playbooks/roles/gen_nodes/tasks/main.yml b/playbooks/roles/gen_nodes/tasks/main.yml
index 2f5c48b6cdcf..92cf5b3065a2 100644
--- a/playbooks/roles/gen_nodes/tasks/main.yml
+++ b/playbooks/roles/gen_nodes/tasks/main.yml
@@ -385,6 +385,19 @@
with_items: "{{ guestfs_nodes }}"
when: kdevops_enable_guestfs
+- name: Generate XML files for the libvirt guests for PCI passthrough for guestfs
+ delegate_to: localhost
+ run_once: true
+ shell: |
+ set -o pipefail
+ {{ role_path }}/python/gen_pcie_passthrough_guestfs_xml.py
+ args:
+ executable: /bin/bash
+ chdir: "{{ topdir_path }}"
+ when:
+ - kdevops_enable_guestfs
+ - pcie_passthrough_enable|bool
+
- name: Generate XML files for the libvirt guests
vars:
hostname: "{{ item.name }}"
diff --git a/scripts/bringup_guestfs.sh b/scripts/bringup_guestfs.sh
index 2b5b38571265..b02c3ef3ba4c 100755
--- a/scripts/bringup_guestfs.sh
+++ b/scripts/bringup_guestfs.sh
@@ -135,6 +135,10 @@ do
fi
virsh define $GUESTFSDIR/$name/$name.xml
+ for xml in $GUESTFSDIR/$name/pcie_passthrough_*.xml; do
+ echo "Doing PCI-E passthrough for device $xml"
+ virsh attach-device $name $xml --config
+ done
virsh start $name
if [[ $? -ne 0 ]]; then
echo "Failed to start $name"
--
2.43.0
reply other threads:[~2024-03-08 2:53 UTC|newest]
Thread overview: [no followups] expand[flat|nested] mbox.gz Atom feed
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20240308025338.1768534-1-mcgrof@kernel.org \
--to=mcgrof@kernel.org \
--cc=a.manzanares@samsung.com \
--cc=dave@stgolabs.net \
--cc=fan.ni@samsung.com \
--cc=jim.harris@samsung.com \
--cc=kdevops@lists.linux.dev \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).