From: Luis Chamberlain <mcgrof@kernel.org>
To: "Richard W.M. Jones" <rjones@redhat.com>
Cc: guestfs@lists.libguestfs.org, kdevops@lists.linux.dev,
Ross Vandegrift <ross@kallisti.us>
Subject: Re: Setting up guestfs mirrors - v2
Date: Fri, 29 Mar 2024 10:52:01 -0700 [thread overview]
Message-ID: <Zgb_wbIep4yAXsjC@bombadil.infradead.org> (raw)
In-Reply-To: <20240329091752.GS7636@redhat.com>
+ cc kdevops and Ross from the debian side.
On Fri, Mar 29, 2024 at 09:17:52AM +0000, Richard W.M. Jones wrote:
> > On Thu, Mar 28, 2024 at 09:33:50AM -0700, Luis Chamberlain wrote:
> > > [ v2 as I don't think the v1 email got through to guestfs list ]
> > >
> > > Thanks to the work by Jeff Layton, we've embraced guestfs on kdevops
> > > [0] as a replacement to vagrant. To save bandwidth kdevops lets you
> > > set up mirrors for all git trees we need, it would be nice to make it
> > > easier for folks to also setup mirrors for guestfs base images, the
> > > base images are here:
> > >
> > > https://builder.libguestfs.org/
> > >
> > > I take it we'd just add say
> > > $XDG_CONFIG_HOME/virt-builder/repos.d/kdevops-local-mirror.conf with
> > > the new URL where we established the mirror, is that correct?
>
> Yes.
Neat!
> > > I was looking to see if there was an rsync mirror for the main URL
> > > where all the images are above, or is there a better way to get the
> > > latest updates?
>
> Ideally we'd like to not host the images at all, but getting distros
> to host them instead has not been very successful (except for SUSE).
It makes sense, it would put all the burden on just one server, and the
licensing considerations are already figured out by distros
as well so might as well leverage that. So I see four for distros
to host them:
1) decentralizes
2) distros already produce cloud images, having a guestfs image
should be trivial to produce, just as current vagrant images
3) licensing is already figured out
4) since some distros also provie updated images than the base,
as with debian testing vagrant images, they can also provide
more up to date guestfs images as well, reducing the amount of
time it takes to have an up to date system.
If distros do start shipping guestfs images, then eventually
we want to update upstream virt-builder with a respective:
$XDG_CONFIG_HOME/virt-builder/repos.d/$distro.conf
That would allo $distro's source spelled out for their own updates.
A good example as you hinted was that opensuse already does exactly
this, so we have:
cat /etc/virt-builder/repos.d/opensuse.conf
[opensuse.org]
uri=http://download.opensuse.org/repositories/Virtualization:/virt-builder-images/images/index
gpgkey=file:///etc/virt-builder/repos.d/opensuse.gpg
> > > Also, to save even more time with images, it would be useful if we had
> > > base images getting updates a bit more regularly so that the step to
> > > apt-get upgrade after bringup won't take too long by using the latest
> > > base images. It seems this is possible with customer images, but it
> > > would be nicer if we had this as a general thing. Any ideas to support
> > > that?
>
> Yes, we'd like Debian to generate their own images! Actually
> they don't even need to generate images at all, just host
> the index file.
Great! Let's all start helping to persuade each distro to do this then.
In the meantime, to help alleviate the load off of builder.libguestfs.org
it would be good to at laest host a few mirrors, maybe some public maybe
some not public. To help with this I'd like some clarifications.
I just discovered and ran virt-builder --cache-all-templates, which
downloads all the 29 GiB of base images onto my ~/.cache/virt-builder/
but I don't see the index file there, so this does not seem to be
an equivalent of what would be an rsync of the data on
https://builder.libguestfs.org/ so I don't think the contents of this
directory could be used without postprocessing of the same data on the
builder page. This is not an issue if all one wants is to just mirror
the data locally, but say if one wanted to *host* a exact mirror of
what you have in the URL it would need some post processing. For
instance the above URL has a XZ compressed debian-12.xz while the command
virt-builder --cache-all-templates gets me an XZ compressed file
~/.cache/virt-builder/debian-12.x86_64.1 decompressing both and
sha1suming confirms its the same file with an sha1sum of
6fc19d34398c2c57ebd396c080d63531238f3ac43b61c66ae167903188404bf44de29b677085b9419f906a30cc720afc3fe43af7e09d970beadd8b673a06ca86
and with it compressed as reflected in the public index file:
f1c58b53b7d25691ccbd2c3eb97f692985e8a035247dee3ed8e024bbae64574620ace75dbb57207064739f5e4b317c6ad6d6d518d777c719441b06b1634fe07e
Is there a better way to host an exact copy of what's in
https://builder.libguestfs.org and what would a conf that
does this look like?
Let's say we want to implement private or public mirrors to help,
once we figure out how to mirror mirror same data, and we dump it on
page say on foobar.org would we want to add to a system:
/etc/virt-builder/repos.d/foobar.conf
[foobar.org]
uri=http://foobar.org/download/builder/index.asc
gpgkey=file:///etc/xdg/virt-builder/repos.d/libguestfs.gpg
Luis
next parent reply other threads:[~2024-03-29 17:52 UTC|newest]
Thread overview: 3+ messages / expand[flat|nested] mbox.gz Atom feed top
[not found] <ZgWtBX5dZEtqtdYH@bombadil.infradead.org>
[not found] ` <20240329091752.GS7636@redhat.com>
2024-03-29 17:52 ` Luis Chamberlain [this message]
2024-04-03 21:33 ` Setting up guestfs mirrors - v2 Richard W.M. Jones
2024-03-28 16:33 Luis Chamberlain
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=Zgb_wbIep4yAXsjC@bombadil.infradead.org \
--to=mcgrof@kernel.org \
--cc=guestfs@lists.libguestfs.org \
--cc=kdevops@lists.linux.dev \
--cc=rjones@redhat.com \
--cc=ross@kallisti.us \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).