From: Chuck Lever <cel@kernel.org>
To: kdevops@lists.linux.dev
Cc: Michael Christie <michael.christie@oracle.com>,
Chuck Lever <chuck.lever@oracle.com>
Subject: [PATCH RFC 4/6] nfsd: Provision an iSCSI initiator on the kdevops NFS server
Date: Wed, 15 May 2024 16:49:27 -0400 [thread overview]
Message-ID: <171580616792.111411.5317905117919462327.stgit@oracle-102.chuck.lever.oracle.com.nfsv4.dev> (raw)
In-Reply-To: <171580595774.111411.12296721249184382497.stgit@oracle-102.chuck.lever.oracle.com.nfsv4.dev>
From: Chuck Lever <chuck.lever@oracle.com>
Enable NFS exports to be placed on iSCSI LUNs. This is a common
deployment practice for pNFS block layout.
With this commit, NFSD configuration now gives the option of placing
the physical storage for exports locally (as is done currently) or
on an iSCSI LUN. Then all exported data on the NFS server will
reside either in a local volume group on the NFS server, or on an
external iSCSI LUN. Later we might want to add other storage
locations, like adding NVMe.
Note that although using a LUN can support any file system type
that resides on durable storage, only XFS supports the pNFS block
Cc: Michael Christie <michael.christie@oracle.com>
Signed-off-by: Chuck Lever <chuck.lever@oracle.com>
---
kconfigs/Kconfig.nfsd | 57 +++++++++++++------
playbooks/roles/nfsd/defaults/main.yml | 3 +
playbooks/roles/nfsd/tasks/main.yml | 17 ++++--
playbooks/roles/nfsd_add_export/defaults/main.yml | 3 +
playbooks/roles/nfsd_add_export/tasks/main.yml | 8 +++
.../roles/nfsd_add_export/tasks/storage/iscsi.yml | 61 ++++++++++++++++++++
scripts/nfsd.Makefile | 8 +++
7 files changed, 136 insertions(+), 21 deletions(-)
create mode 100644 playbooks/roles/nfsd_add_export/tasks/storage/iscsi.yml
diff --git a/kconfigs/Kconfig.nfsd b/kconfigs/Kconfig.nfsd
index dec98c1e964f..d071f5fba278 100644
--- a/kconfigs/Kconfig.nfsd
+++ b/kconfigs/Kconfig.nfsd
@@ -11,23 +11,6 @@ if KDEVOPS_SETUP_NFSD
menu "Configure the kernel NFS server"
-config NFSD_EXPORT_DEVICE_PREFIX
- string "The device prefix to use for LVM PVs"
- default "/dev/disk/by-id/nvme-QEMU_NVMe_Ctrl_kdevops" if LIBVIRT && LIBVIRT_EXTRA_STORAGE_DRIVE_NVME
- default "/dev/disk/by-id/virtio-kdevops" if LIBVIRT && LIBVIRT_EXTRA_STORAGE_DRIVE_VIRTIO
- default "/dev/disk/by-id/ata-QEMU_HARDDISK_kdevops" if LIBVIRT && LIBVIRT_EXTRA_STORAGE_DRIVE_IDE
- default ""
- help
- To set up nfsd for testing, we give it filesystems to export. This string
- will be the prefix for the block devices used as PVs for LVM.
-
-config NFSD_EXPORT_DEVICE_COUNT
- int "Number of devices to add as LVM PVs"
- default 3 if LIBVIRT
- help
- The number of disk devices to dedicate as LVM PVs. In general, we
- avoid using device index 0 as that is used for /data.
-
choice
prompt "Type of filesystem to export"
default NFSD_EXPORT_FSTYPE_BTRFS
@@ -83,6 +66,46 @@ config NFSD_LEASE_TIME
and lock state. But a shorter lease time helps certain tests
complete faster.
+choice
+ prompt "Local or external physical storage"
+ default NFSD_EXPORT_STORAGE_LOCAL
+
+config NFSD_EXPORT_STORAGE_LOCAL
+ bool "Local"
+ help
+ Exported file systems will reside on physical storage
+ local to the NFS server itself.
+
+config NFSD_EXPORT_STORAGE_ISCSI
+ bool "iSCSI"
+ help
+ Exported file systems will reside on physical storage
+ located on a separate target node and accessed via
+ iSCSI.
+
+endchoice
+
+if NFSD_EXPORT_STORAGE_LOCAL
+
+config NFSD_EXPORT_DEVICE_PREFIX
+ string "The device prefix to use for LVM PVs"
+ default "/dev/disk/by-id/nvme-QEMU_NVMe_Ctrl_kdevops" if LIBVIRT && LIBVIRT_EXTRA_STORAGE_DRIVE_NVME
+ default "/dev/disk/by-id/virtio-kdevops" if LIBVIRT && LIBVIRT_EXTRA_STORAGE_DRIVE_VIRTIO
+ default "/dev/disk/by-id/ata-QEMU_HARDDISK_kdevops" if LIBVIRT && LIBVIRT_EXTRA_STORAGE_DRIVE_IDE
+ default ""
+ help
+ To set up nfsd for testing, we give it filesystems to export. This string
+ will be the prefix for the block devices used as PVs for LVM.
+
+config NFSD_EXPORT_DEVICE_COUNT
+ int "Number of devices to add as LVM PVs"
+ default 3 if LIBVIRT
+ help
+ The number of disk devices to dedicate as LVM PVs. In general, we
+ avoid using device index 0 as that is used for /data.
+
+endif
+
endmenu
endif
diff --git a/playbooks/roles/nfsd/defaults/main.yml b/playbooks/roles/nfsd/defaults/main.yml
index 7fb98c25c45d..271d2d1d8912 100644
--- a/playbooks/roles/nfsd/defaults/main.yml
+++ b/playbooks/roles/nfsd/defaults/main.yml
@@ -8,4 +8,7 @@ nfsd_export_label: "export"
nfsd_export_fs_opts: ""
nfsd_lease_time: "90"
+nfsd_export_storage_local: false
+nfsd_export_storage_iscsi: false
+
kdevops_krb5_enable: false
diff --git a/playbooks/roles/nfsd/tasks/main.yml b/playbooks/roles/nfsd/tasks/main.yml
index fe5f6919864d..63388f857627 100644
--- a/playbooks/roles/nfsd/tasks/main.yml
+++ b/playbooks/roles/nfsd/tasks/main.yml
@@ -22,14 +22,21 @@
group: root
mode: 0644
+- name: Set up an iSCSI initiator
+ ansible.builtin.include_role:
+ name: iscsi
+ tasks_from: add_initiator
+ when:
+ - nfsd_export_storage_iscsi|bool
+
- name: Build string of devices to use as PVs
set_fact:
nfsd_lvm_pvs: "{{ nfsd_lvm_pvs + [ nfsd_export_device_prefix + item|string ] }}"
with_items: "{{ range(1, nfsd_export_device_count + 1) }}"
-
-- name: Print the PV list
- ansible.builtin.debug:
- var: nfsd_lvm_pvs
+ loop_control:
+ label: "Physical volume: {{ nfsd_export_device_prefix + item|string }}"
+ when:
+ - nfsd_export_storage_local|bool
- name: Create a new LVM VG
become: yes
@@ -38,6 +45,8 @@
community.general.lvg:
vg: "exports"
pvs: "{{ nfsd_lvm_pvs | join(',') }}"
+ when:
+ - nfsd_export_storage_local|bool
- name: Create {{ nfsd_export_path }}
become: yes
diff --git a/playbooks/roles/nfsd_add_export/defaults/main.yml b/playbooks/roles/nfsd_add_export/defaults/main.yml
index d4cea833f36d..87295e8f2c91 100644
--- a/playbooks/roles/nfsd_add_export/defaults/main.yml
+++ b/playbooks/roles/nfsd_add_export/defaults/main.yml
@@ -3,3 +3,6 @@
export_user: "root"
export_group: "root"
export_mode: "u=rwx,g=rwx,o=rwxt"
+
+nfsd_export_storage_local: false
+nfsd_export_storage_iscsi: false
diff --git a/playbooks/roles/nfsd_add_export/tasks/main.yml b/playbooks/roles/nfsd_add_export/tasks/main.yml
index f81fe1e95568..d72a721c9085 100644
--- a/playbooks/roles/nfsd_add_export/tasks/main.yml
+++ b/playbooks/roles/nfsd_add_export/tasks/main.yml
@@ -3,6 +3,14 @@
ansible.builtin.include_tasks:
file: storage/local.yml
when:
+ - nfsd_export_storage_local|bool
+ - export_fstype != "tmpfs"
+
+- name: Add an iSCSI LUN for the new export
+ ansible.builtin.include_tasks:
+ file: storage/iscsi.yml
+ when:
+ - nfsd_export_storage_iscsi|bool
- export_fstype != "tmpfs"
- name: Add a tmpfs for the new export
diff --git a/playbooks/roles/nfsd_add_export/tasks/storage/iscsi.yml b/playbooks/roles/nfsd_add_export/tasks/storage/iscsi.yml
new file mode 100644
index 000000000000..098f6fa73ab8
--- /dev/null
+++ b/playbooks/roles/nfsd_add_export/tasks/storage/iscsi.yml
@@ -0,0 +1,61 @@
+---
+- name: Create an iSCSI LUN for the new export
+ vars:
+ iscsi_add_devname: "{{ export_volname }}"
+ iscsi_add_size: "{{ export_size }}"
+ ansible.builtin.include_role:
+ name: iscsi
+ tasks_from: add_lun
+
+- name: Rescan iSCSI LUNs on the NFS server
+ become: true
+ become_flags: 'su - -c'
+ become_method: ansible.builtin.sudo
+ delegate_to: "{{ server_host }}"
+ community.general.open_iscsi:
+ rescan: true
+
+- name: Rescan iSCSI LUNs on the target node
+ become: true
+ become_flags: 'su - -c'
+ become_method: ansible.builtin.sudo
+ community.general.open_iscsi:
+ rescan: true
+
+- name: Enumerate available SCSI devices on the NFS server
+ become: true
+ become_flags: 'su - -c'
+ become_method: ansible.builtin.sudo
+ delegate_to: "{{ server_host }}"
+ ansible.builtin.command:
+ cmd: "lsblk -Snp -o MODEL,NAME"
+ register: lsblk_output
+ changed_when: false
+
+- name: Select the device that matches {{ export_volname }}
+ ansible.builtin.set_fact:
+ iscsi_device: "{{ item | split() | last }}"
+ loop: "{{ lsblk_output.stdout_lines }}"
+ when: export_volname in item
+
+- name: Format new iSCSI LUN for {{ export_fstype }}
+ become: true
+ become_flags: 'su - -c'
+ become_method: ansible.builtin.sudo
+ delegate_to: "{{ server_host }}"
+ community.general.filesystem:
+ fstype: "{{ export_fstype }}"
+ dev: "{{ iscsi_device }}"
+
+- name: Mount iSCSI LUN under {{ nfsd_export_path }}
+ become: true
+ become_flags: 'su - -c'
+ become_method: ansible.builtin.sudo
+ delegate_to: "{{ server_host }}"
+ throttle: 1 # running this in parallel can clobber fstab changes
+ ansible.posix.mount:
+ path: "{{ nfsd_export_path }}/{{ export_volname }}"
+ src: "{{ iscsi_device }}"
+ fstype: "{{ export_fstype }}"
+ opts: "_netdev"
+ state: mounted
diff --git a/scripts/nfsd.Makefile b/scripts/nfsd.Makefile
index 3accc87901c1..ebdbb4dd1e5a 100644
--- a/scripts/nfsd.Makefile
+++ b/scripts/nfsd.Makefile
@@ -1,7 +1,15 @@
ifeq (y,$(CONFIG_KDEVOPS_SETUP_NFSD))
+ifeq (y,$(CONFIG_NFSD_EXPORT_STORAGE_LOCAL))
+NFSD_EXTRA_ARGS += nfsd_export_storage_local=true
NFSD_EXTRA_ARGS += nfsd_export_device_prefix='$(subst ",,$(CONFIG_NFSD_EXPORT_DEVICE_PREFIX))'
NFSD_EXTRA_ARGS += nfsd_export_device_count='$(subst ",,$(CONFIG_NFSD_EXPORT_DEVICE_COUNT))'
+endif
+
+ifeq (y,$(CONFIG_NFSD_EXPORT_STORAGE_ISCSI))
+NFSD_EXTRA_ARGS += nfsd_export_storage_iscsi=true
+endif
+
NFSD_EXTRA_ARGS += nfsd_export_fstype='$(subst ",,$(CONFIG_NFSD_EXPORT_FSTYPE))'
NFSD_EXTRA_ARGS += nfsd_export_path='$(subst ",,$(CONFIG_NFSD_EXPORT_PATH))'
NFSD_EXTRA_ARGS += nfsd_threads=$(CONFIG_NFSD_THREADS)
next prev parent reply other threads:[~2024-05-15 20:49 UTC|newest]
Thread overview: 7+ messages / expand[flat|nested] mbox.gz Atom feed top
2024-05-15 20:49 [PATCH RFC 0/6] Introduce iSCSI target for kdevops Chuck Lever
2024-05-15 20:49 ` [PATCH RFC 1/6] nfsd_add_export: Move storage allocation to separate YML files Chuck Lever
2024-05-15 20:49 ` [PATCH RFC 2/6] Shorten the names of devices where exports reside Chuck Lever
2024-05-15 20:49 ` [PATCH RFC 3/6] iscsi: Provision a target node to host iSCSI LUNs Chuck Lever
2024-05-15 20:49 ` Chuck Lever [this message]
2024-05-15 20:49 ` [PATCH RFC 5/6] nfsd_add_export: Enable pnfs on capable exports Chuck Lever
2024-05-15 20:49 ` [PATCH RFC 6/6] gitr: Enable testing with pNFS block layouts Chuck Lever
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=171580616792.111411.5317905117919462327.stgit@oracle-102.chuck.lever.oracle.com.nfsv4.dev \
--to=cel@kernel.org \
--cc=chuck.lever@oracle.com \
--cc=kdevops@lists.linux.dev \
--cc=michael.christie@oracle.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).