From: Luis Chamberlain <mcgrof@kernel.org>
To: Scott Mayhew <smayhew@redhat.com>,
"Richard W.M. Jones" <rjones@redhat.com>
Cc: kdevops@lists.linux.dev
Subject: Re: [PATCH 05/10] guestfs: add initial debian trixie support with custom URLs
Date: Sat, 11 May 2024 16:46:48 -0700 [thread overview]
Message-ID: <ZkADaA8omNGuOpCh@bombadil.infradead.org> (raw)
In-Reply-To: <Zju2wu1IZYd1hmA7@aion>
On Wed, May 08, 2024 at 01:30:42PM -0400, Scott Mayhew wrote:
> On Tue, 07 May 2024, Luis Chamberlain wrote:
>
> > debian does not yet provide an index file for virt-builder, but we
> > have URLS with qcow2 and raw files. Using them in guestfs is actually
> > not quite trivial. So we do the handy work to enable others to also
> > use custom URLs and build a virt-builder local source and index file
> > for you. All we need really, is to check the checksums.
> >
> > Sadly this does not yet work, as it seems we get stuck on the grub
> > prompt for some reason. This needs some more investigation.
>
> Yeah when I run 'make bringup' the consoles on all the guests are stuck
> on "Booting `Debian GNU/Linux'".
Is that with Trixie or also with y9our own custom ISO?
> I tried manually running virt-builder + virt-install using the same
> image and I get the same result. Granted, I've only ever used the
> images in the libguestfs repos or images that I've built myself, so I
> could be doing something wrong.
I'm in hopes we can figure this out, there is likely something special
done on the images hosted by libguestfs.org which regular nocloud images
like debian's trixie is not doing:
https://cloud.debian.org/images/cloud/trixie/daily/latest/debian-13-genericcloud-amd64-daily.raw
Once we figure that out, we should be able to easily do that as a post
up with virt-builder.
Richard, any hints on what likely should help fix boot?
> > diff --git a/kconfigs/Kconfig.guestfs b/kconfigs/Kconfig.guestfs
> > index 5838522908e8..5839fbedfd08 100644
> > --- a/kconfigs/Kconfig.guestfs
> > +++ b/kconfigs/Kconfig.guestfs
> > @@ -1,5 +1,30 @@
> > if GUESTFS
> >
> > +config GUESTFS_HAS_CUSTOM_RAW_IMAGE
> > + bool
>
> It might be useful for these variables to have prompts so they show up
> in 'make menuconfig'.
That's a one line change, it just needs a description, for now its auto
because there is no custom raw image URL option, what you descrie seems
for us to want such a new entry and option. So sure, but I think that
can be done in subsequent patch?
> I have a whole bunch of workflows running for
> RHEL8 and RHEL9 nightly builds... they start with 8.9 or 9.3
> images
I see, yes, we wouldn't need to let you enable GUESTFS_HAS_CUSTOM_RAW_IMAGE
but rather waht we want is just a drop down menu for distro to let you
select "custom", that way you could use a set of defconfigs which would
use the custom image URL, when that is enabled it selects
GUESTFS_HAS_CUSTOM_RAW_IMAGE.
> and use the CONFIG_KDEVOPS_CUSTOM_YUM_REPOFILE to update them to
> the latest bits.
My patch after this "guestfs: add support to infer host distro
mirrororing optimizations" helps to infer when we want to copy a
custom sources list, based on a host's hop count. What you describe
seems to beg for another auto-infererence which could be based on
some target directory and based on on the custom URL. That could be
added and then you wouldn't even need to make any custom .config other
than having a few defconfigs for each distro point release.
> If I could build my own images and dump them in a
> directory without having to worry about updating the index file that
> would probably speed things up a bit.
I think this is possible, we could just look for directory we add to
.gitignore, and if the user has it as a symlink, we leverage that as
a key indicator for "hunting season open for custom images, please
populate my Kconfig with these other custom options".
> > +
> > +config GUESTFS_HAS_CUSTOM_RAW_IMAGE_URL
> > + bool
> > +
> > +config GUESTFS_CUSTOM_RAW_IMAGE_URL
> > + depends on GUESTFS_HAS_CUSTOM_RAW_IMAGE
> > + depends on GUESTFS_HAS_CUSTOM_RAW_IMAGE_URL
> > + string
> > + default "https://cloud.debian.org/images/cloud/trixie/daily/latest/debian-13-generic-amd64-daily" if GUESTFS_DEBIAN_TRIXIE_GENERIC_AMD64
> > + default "https://cloud.debian.org/images/cloud/trixie/daily/latest/debian-13-genericcloud-amd64-daily" if GUESTFS_DEBIAN_TRIXIE_GENERIC_CLOUD_AMD64
> > + default "https://cloud.debian.org/images/cloud/trixie/daily/latest/debian-13-nocloud-amd64-daily" if GUESTFS_DEBIAN_TRIXIE_NOCLOUD_AMD64
>
> You need '.raw' at the end of the URL, right?
Oops yes sorry.
> > diff --git a/scripts/bringup_guestfs.sh b/scripts/bringup_guestfs.sh
> > index f90ed499051b..514a26a60436 100755
> > --- a/scripts/bringup_guestfs.sh
> > +++ b/scripts/bringup_guestfs.sh
> > @@ -30,8 +30,135 @@ GUESTFSDIR="${TOPDIR}/guestfs"
> > OS_VERSION=${CONFIG_VIRT_BUILDER_OS_VERSION}
> > BASE_IMAGE_DIR="${STORAGEDIR}/base_images"
> > BASE_IMAGE="${BASE_IMAGE_DIR}/${OS_VERSION}.raw"
> > +
> > +build_custom_source()
> > +{
> > + SOURCE_TMP=$(mktemp)
> > + cat <<_EOT >$SOURCE_TMP
> > +[local]
> ^^^
> You probably want this to be unique based on the flavor. If you switch
> flavors, then you're going to wind up with multiple repo configs with the
> same repo id and virt-builder will only pick up the last one it reads...
> the end result being that it probably won't find the template you want
> to install.
So this is for the source, not the index, the source is local, can we
not have multiple arbitrary sources with the same name? The indexes for
each file would be different. Here I just wanted to emphasize that the
source is local.
>
> > +uri=file:///${CUSTOM_INDEX}
> > +proxy=off
> > +_EOT
See the source, points to a custom index, and so we'd have multiple
local sources, the index files do change per flavor.
But certainly, it would be good to get this right to make this scale.
For now to start off with trixie, and later to scale to whatever custom
setup we need for random builds.
Thoughts / advice?
Anyway, since this still has a big fat warning as "this doesn't work
yet", I've made the changes required so far and pushed these patches in,
we can make changes to scale this, now that at laest its easier to test
these things.
Luis
next prev parent reply other threads:[~2024-05-11 23:46 UTC|newest]
Thread overview: 17+ messages / expand[flat|nested] mbox.gz Atom feed top
2024-05-08 6:50 [PATCH 00/10] guestfs: custom image + mirroring sources.list heuristic Luis Chamberlain
2024-05-08 6:50 ` [PATCH 01/10] guestfs: move debian options before image names Luis Chamberlain
2024-05-08 6:50 ` [PATCH 02/10] guestfs: modify grub prompt before first bring up Luis Chamberlain
2024-05-08 6:50 ` [PATCH 03/10] guestfs: set default root password Luis Chamberlain
2024-05-08 6:50 ` [PATCH 04/10] guestfs: check for virt-builder failure Luis Chamberlain
2024-05-08 6:50 ` [PATCH 05/10] guestfs: add initial debian trixie support with custom URLs Luis Chamberlain
2024-05-08 17:30 ` Scott Mayhew
2024-05-11 23:46 ` Luis Chamberlain [this message]
2024-05-13 18:28 ` Richard W.M. Jones
2024-05-13 20:50 ` Luis Chamberlain
2024-05-13 20:55 ` Scott Mayhew
2024-05-14 12:04 ` Luis Chamberlain
2024-05-08 6:50 ` [PATCH 06/10] guestfs: add support to infer host distro mirrororing optimizations Luis Chamberlain
2024-05-08 6:50 ` [PATCH 07/10] guestfs: move rhel activation to its own helper Luis Chamberlain
2024-05-08 6:50 ` [PATCH 08/10] guestfs: move copying yum repo to its own routine Luis Chamberlain
2024-05-08 6:50 ` [PATCH 09/10] guestfs: move pre-install customizations " Luis Chamberlain
2024-05-08 6:50 ` [PATCH 10/10] guestfs: move debian pre-install hacks " Luis Chamberlain
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=ZkADaA8omNGuOpCh@bombadil.infradead.org \
--to=mcgrof@kernel.org \
--cc=kdevops@lists.linux.dev \
--cc=rjones@redhat.com \
--cc=smayhew@redhat.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).