* [PATCH v9 0/3] pkg-database and systemd-sysext image
@ 2024-05-15 22:34 Johannes Schneider
2024-05-15 22:34 ` [PATCH v9 1/3] image.bbclass/rootfs: archive and deploy package database Johannes Schneider
` (3 more replies)
0 siblings, 4 replies; 8+ messages in thread
From: Johannes Schneider @ 2024-05-15 22:34 UTC (permalink / raw
To: openembedded-core, richard.purdie, alex.kanavin,
alexandre.belloni
Cc: Johannes Schneider
systemd-sysext allows to overlay another image (or multiple) ontop of
a "base-image" = the current rootfs, via the use of overlayfs; to add
tools and features meant for development purposes.
To quote the documentation on systemd-sysext:
" ...addition in order to make debugging/development easier). System
extension images should not be misunderstood as a generic software
packaging framework, ..."
To build a lean image, that only holds packages that are not already
part of the base-image, a snapshot of the package-database is taken
after the installation of the base-rootfs is done, and picked up again
when collecting the rootfs of such a extension image.
with all this in place an example usage could look like this:
some-core-image.bb
inherit core-image
IMAGE_GEN_PKGDBFS = "1"
extending-image.bb
inherit image-sysext
IMAGE_FSTYPES = "squashfs"
IMAGE_BASE_PKGDB = "some-core-image"
# the above pointing at a package-db similar to:
# build/deploy/images/$MACHINE/some-core-image-$MACHINE-20240210172305-pkgdb.rootfs.tar.gz
then on the device, running some-core-image, with the extension image placed at FN:
$> ln -s "$FN" /run/extensions/$(basename $FN).raw
$> systemd-sysext list
$> SYSTEMD_LOG_LEVEL=debug systemd-sysext merge
As long as the VERSION_ID of the extension image matches the os-release
in the base image, the above commands return sucessfully;
for details on the compativility check see the docs for systemd-sysext.
=========
changes with v2:
rebase from 'kirkstone' onto 'master'
changes with v3:
incorporate review suggestions for simplification
add task dependency handling
add oe-selftest for the pkgdb handling
add variable documentation and
some more comments, and examples in the commit-msg
changes with v4:
rebase onto 'master' => no functional changes
fixed patchtest findings
changes with v5:
rebase onto 'master'
add '.sysext' to the deployed symlink name
sidenote on the tests and autobuilder failure: run locally they succeed, e.g.:
#> oe-selftest --verbose -r imagefeatures.ImageFeatures.test_image_gen_pkgdbfs
changes with v6:
tests: restructure to call 'bitbake' only once in the testcase itself
(in hopes of solving the autobuilder problem; local test runs succeed)
changes with v7:
tests: undo the restructuring of done in v6, in favour of explicitly forcing steps:
'-c rootfs -f'. since concurrency of the autobuilder, and reusing artifacts/sstate-caching is probably the issue?
changes with v8:
tests: fixed missing feature=PACKAGE_CLASSES during 'test_image_gen_pkgdbfs'
which is now split into three, one for each of the three available packagemanager: ipk, deb, rpm
changes with v9:
tests: fix singular failure in 'oe-selftest-armhost'
which was due to the built image not containing a 'kernel-image...', which the gen-pkgdb sanity
check expected, but 'packagegroup-core-boot' instead
^ permalink raw reply [flat|nested] 8+ messages in thread
* [PATCH v9 1/3] image.bbclass/rootfs: archive and deploy package database
2024-05-15 22:34 [PATCH v9 0/3] pkg-database and systemd-sysext image Johannes Schneider
@ 2024-05-15 22:34 ` Johannes Schneider
2024-05-15 22:34 ` [PATCH v9 2/3] image.bbclass/rootfs: set and unpack package-database Johannes Schneider
` (2 subsequent siblings)
3 siblings, 0 replies; 8+ messages in thread
From: Johannes Schneider @ 2024-05-15 22:34 UTC (permalink / raw
To: openembedded-core, richard.purdie, alex.kanavin,
alexandre.belloni
Cc: Johannes Schneider
archive the package database after the rootfs has been put together as
*rootfs-pkdbfs.tar.gz, and put it into the deploy folder.
This creates a snapshot of the package mangers state at the point in
time when all dependencies have been resolved and installed; which
could be used by "extension images" to built upon.
Signed-off-by: Johannes Schneider <johannes.schneider@leica-geosystems.com>
---
meta/classes-recipe/image.bbclass | 44 ++++-
meta/classes-recipe/image_types.bbclass | 1 +
meta/conf/documentation.conf | 1 +
meta/lib/oe/package_manager/deb/rootfs.py | 1 +
meta/lib/oe/package_manager/ipk/rootfs.py | 1 +
meta/lib/oe/package_manager/rpm/rootfs.py | 1 +
meta/lib/oe/rootfs.py | 20 +++
meta/lib/oeqa/selftest/cases/imagefeatures.py | 159 ++++++++++++++++++
8 files changed, 227 insertions(+), 1 deletion(-)
diff --git a/meta/classes-recipe/image.bbclass b/meta/classes-recipe/image.bbclass
index 28be6c6362..3ccaaa17b8 100644
--- a/meta/classes-recipe/image.bbclass
+++ b/meta/classes-recipe/image.bbclass
@@ -42,6 +42,9 @@ IMAGE_FEATURES ?= ""
IMAGE_FEATURES[type] = "list"
IMAGE_FEATURES[validitems] += "debug-tweaks read-only-rootfs read-only-rootfs-delayed-postinsts stateless-rootfs empty-root-password allow-empty-password allow-root-login serial-autologin-root post-install-logging overlayfs-etc"
+# Generate snapshot of the package database?
+IMAGE_GEN_PKGDBFS ?= "0"
+
# Generate companion debugfs?
IMAGE_GEN_DEBUGFS ?= "0"
@@ -131,7 +134,8 @@ def rootfs_variables(d):
'IMAGE_ROOTFS_MAXSIZE','IMAGE_NAME','IMAGE_LINK_NAME','IMAGE_MANIFEST','DEPLOY_DIR_IMAGE','IMAGE_FSTYPES','IMAGE_INSTALL_COMPLEMENTARY','IMAGE_LINGUAS', 'IMAGE_LINGUAS_COMPLEMENTARY', 'IMAGE_LOCALES_ARCHIVE',
'MULTILIBRE_ALLOW_REP','MULTILIB_TEMP_ROOTFS','MULTILIB_VARIANTS','MULTILIBS','ALL_MULTILIB_PACKAGE_ARCHS','MULTILIB_GLOBAL_VARIANTS','BAD_RECOMMENDATIONS','NO_RECOMMENDATIONS',
'PACKAGE_ARCHS','PACKAGE_CLASSES','TARGET_VENDOR','TARGET_ARCH','TARGET_OS','OVERRIDES','BBEXTENDVARIANT','FEED_DEPLOYDIR_BASE_URI','INTERCEPT_DIR','USE_DEVFS',
- 'CONVERSIONTYPES', 'IMAGE_GEN_DEBUGFS', 'ROOTFS_RO_UNNEEDED', 'IMGDEPLOYDIR', 'PACKAGE_EXCLUDE_COMPLEMENTARY', 'REPRODUCIBLE_TIMESTAMP_ROOTFS', 'IMAGE_INSTALL_DEBUGFS']
+ 'CONVERSIONTYPES', 'IMAGE_GEN_PKGDBFS', 'IMAGE_GEN_DEBUGFS', 'ROOTFS_RO_UNNEEDED', 'IMGDEPLOYDIR', 'PACKAGE_EXCLUDE_COMPLEMENTARY', 'REPRODUCIBLE_TIMESTAMP_ROOTFS',
+ 'IMAGE_INSTALL_DEBUGFS']
variables.extend(rootfs_command_variables(d))
variables.extend(variable_depends(d))
return " ".join(variables)
@@ -337,6 +341,17 @@ python do_image_qa_setscene () {
}
addtask do_image_qa_setscene
+def setup_pkgdbfs_variables(d):
+ d.appendVar('IMAGE_ROOTFS', '-pkgdb')
+ if d.getVar('IMAGE_LINK_NAME'):
+ d.appendVar('IMAGE_LINK_NAME', '-pkgdb')
+ d.appendVar('IMAGE_NAME','-pkgdb')
+ d.setVar('IMAGE_FSTYPES', 'tar.gz')
+
+python setup_pkgdbfs () {
+ setup_pkgdbfs_variables(d)
+}
+
def setup_debugfs_variables(d):
d.appendVar('IMAGE_ROOTFS', '-dbg')
if d.getVar('IMAGE_LINK_NAME'):
@@ -381,6 +396,11 @@ python () {
alltypes = d.getVar('IMAGE_FSTYPES').split()
typedeps = {}
+ if d.getVar('IMAGE_GEN_PKGDBFS') == "1":
+ pkgdbfs_fstypes = ['tar.gz']
+ for t in pkgdbfs_fstypes:
+ alltypes.append("pkgdbfs_" + t)
+
if d.getVar('IMAGE_GEN_DEBUGFS') == "1":
debugfs_fstypes = d.getVar('IMAGE_FSTYPES_DEBUGFS').split()
for t in debugfs_fstypes:
@@ -393,6 +413,10 @@ python () {
basetypes[baset]= []
if t not in basetypes[baset]:
basetypes[baset].append(t)
+ pkgdb = ""
+ if t.startswith("pkgdbfs_"):
+ t = t[8:]
+ pkgdb = "pkgdbfs_"
debug = ""
if t.startswith("debugfs_"):
t = t[8:]
@@ -401,6 +425,13 @@ python () {
vardeps.add('IMAGE_TYPEDEP:' + t)
if baset not in typedeps:
typedeps[baset] = set()
+ deps = [pkgdb + dep for dep in deps]
+ for dep in deps:
+ if dep not in alltypes:
+ alltypes.append(dep)
+ _add_type(dep)
+ basedep = _image_base_type(dep)
+ typedeps[baset].add(basedep)
deps = [debug + dep for dep in deps]
for dep in deps:
if dep not in alltypes:
@@ -419,6 +450,7 @@ python () {
maskedtypes = (d.getVar('IMAGE_TYPES_MASKED') or "").split()
maskedtypes = [dbg + t for t in maskedtypes for dbg in ("", "debugfs_")]
+ maskedtypes = [pkgdb + t for t in maskedtypes for pkgdb in ("", "pkgdbfs_")]
for t in basetypes:
vardeps = set()
@@ -430,6 +462,11 @@ python () {
continue
localdata = bb.data.createCopy(d)
+ pkgdb = ""
+ if t.startswith("pkgdbfs_"):
+ setup_pkgdbfs_variables(localdata)
+ pkgdb = "setup_pkgdbfs "
+ realt = t[8:]
debug = ""
if t.startswith("debugfs_"):
setup_debugfs_variables(localdata)
@@ -468,6 +505,8 @@ python () {
for ctype in sorted(ctypes):
if bt.endswith("." + ctype):
type = bt[0:-len(ctype) - 1]
+ if type.startswith("pkgdbfs_"):
+ type = type[8:]
if type.startswith("debugfs_"):
type = type[8:]
# Create input image first.
@@ -509,6 +548,9 @@ python () {
d.setVarFlag(task, 'fakeroot', '1')
d.appendVarFlag(task, 'prefuncs', ' ' + debug + ' set_image_size')
+ if pkgdb:
+ d.appendVarFlag(task, 'prefuncs', ' ' + pkgdb + ' set_image_size')
+
d.prependVarFlag(task, 'postfuncs', 'create_symlinks ')
d.appendVarFlag(task, 'subimages', ' ' + ' '.join(subimages))
d.appendVarFlag(task, 'vardeps', ' ' + ' '.join(vardeps))
diff --git a/meta/classes-recipe/image_types.bbclass b/meta/classes-recipe/image_types.bbclass
index b4a83ae284..602fc26ac3 100644
--- a/meta/classes-recipe/image_types.bbclass
+++ b/meta/classes-recipe/image_types.bbclass
@@ -25,6 +25,7 @@ def imagetypes_getdepends(d):
fstypes = set((d.getVar('IMAGE_FSTYPES') or "").split())
fstypes |= set((d.getVar('IMAGE_FSTYPES_DEBUGFS') or "").split())
+ fstypes |= set('tar.gz') # hardcoded for 'pkgdbfs'
deprecated = set()
deps = set()
diff --git a/meta/conf/documentation.conf b/meta/conf/documentation.conf
index b0591881ba..36aebb59ab 100644
--- a/meta/conf/documentation.conf
+++ b/meta/conf/documentation.conf
@@ -215,6 +215,7 @@ IMAGE_FEATURES[doc] = "The primary list of features to include in an image. Conf
IMAGE_FSTYPES[doc] = "Formats of root filesystem images that you want to have created."
IMAGE_FSTYPES_DEBUGFS[doc] = "Formats of the debug root filesystem images that you want to have created."
IMAGE_GEN_DEBUGFS[doc] = "When set to '1', generate a companion debug object/source filesystem image."
+IMAGE_GEN_PKGDBFS[doc] = "When set to '1', create a snapshot of the package-manager state after the rootfs has been assembled."
IMAGE_INSTALL[doc] = "Specifies the packages to install into an image. Image recipes set IMAGE_INSTALL to specify the packages to install into an image through image.bbclass."
IMAGE_LINGUAS[doc] = "Specifies the list of locales to install into the image during the root filesystem construction process."
IMAGE_NAME[doc] = "The name of the output image files minus the extension."
diff --git a/meta/lib/oe/package_manager/deb/rootfs.py b/meta/lib/oe/package_manager/deb/rootfs.py
index 1e25b64ed9..43107c8663 100644
--- a/meta/lib/oe/package_manager/deb/rootfs.py
+++ b/meta/lib/oe/package_manager/deb/rootfs.py
@@ -178,6 +178,7 @@ class PkgRootfs(DpkgOpkgRootfs):
if self.progress_reporter:
self.progress_reporter.next_stage()
+ self._setup_pkg_db_rootfs(['/var/lib/dpkg'])
self._setup_dbg_rootfs(['/var/lib/dpkg'])
self.pm.fix_broken_dependencies()
diff --git a/meta/lib/oe/package_manager/ipk/rootfs.py b/meta/lib/oe/package_manager/ipk/rootfs.py
index ba93eb62ea..64d9bc7969 100644
--- a/meta/lib/oe/package_manager/ipk/rootfs.py
+++ b/meta/lib/oe/package_manager/ipk/rootfs.py
@@ -319,6 +319,7 @@ class PkgRootfs(DpkgOpkgRootfs):
opkg_lib_dir = self.d.getVar('OPKGLIBDIR')
opkg_dir = os.path.join(opkg_lib_dir, 'opkg')
+ self._setup_pkg_db_rootfs([opkg_dir])
self._setup_dbg_rootfs([opkg_dir])
execute_pre_post_process(self.d, opkg_post_process_cmds)
diff --git a/meta/lib/oe/package_manager/rpm/rootfs.py b/meta/lib/oe/package_manager/rpm/rootfs.py
index 3ba5396320..673006c131 100644
--- a/meta/lib/oe/package_manager/rpm/rootfs.py
+++ b/meta/lib/oe/package_manager/rpm/rootfs.py
@@ -110,6 +110,7 @@ class PkgRootfs(Rootfs):
if self.progress_reporter:
self.progress_reporter.next_stage()
+ self._setup_pkg_db_rootfs(['/etc/rpm', '/etc/rpmrc', '/etc/dnf', '/var/lib/rpm', '/var/cache/dnf', '/var/lib/dnf'])
self._setup_dbg_rootfs(['/etc/rpm', '/etc/rpmrc', '/etc/dnf', '/var/lib/rpm', '/var/cache/dnf', '/var/lib/dnf'])
execute_pre_post_process(self.d, rpm_post_process_cmds)
diff --git a/meta/lib/oe/rootfs.py b/meta/lib/oe/rootfs.py
index 8cd48f9450..6d6e888a30 100644
--- a/meta/lib/oe/rootfs.py
+++ b/meta/lib/oe/rootfs.py
@@ -106,6 +106,26 @@ class Rootfs(object, metaclass=ABCMeta):
def _cleanup(self):
pass
+ def _setup_pkg_db_rootfs(self, package_paths):
+ gen_pkg_db_fs = bb.utils.to_boolean(self.d.getVar('IMAGE_GEN_PKGDBFS'))
+ if not gen_pkg_db_fs:
+ return
+
+ bb.note(" Creating pkg-db rootfs...")
+ try:
+ shutil.rmtree(self.image_rootfs + '-pkgdb')
+ except (FileNotFoundError, NotADirectoryError):
+ pass
+ bb.utils.mkdirhier(self.image_rootfs + "-pkgdb")
+
+ bb.note(" Copying back package database...")
+ for path in package_paths:
+ bb.utils.mkdirhier(self.image_rootfs + os.path.dirname(path))
+ if os.path.isdir(self.image_rootfs + path):
+ shutil.copytree(self.image_rootfs + path, self.image_rootfs + '-pkgdb' + path, symlinks=True)
+ elif os.path.isfile(self.image_rootfs + path):
+ shutil.copyfile(self.image_rootfs + path, self.image_rootfs + '-pkgdb' + path)
+
def _setup_dbg_rootfs(self, package_paths):
gen_debugfs = self.d.getVar('IMAGE_GEN_DEBUGFS') or '0'
if gen_debugfs != '1':
diff --git a/meta/lib/oeqa/selftest/cases/imagefeatures.py b/meta/lib/oeqa/selftest/cases/imagefeatures.py
index dc88c222bd..6724bac0e4 100644
--- a/meta/lib/oeqa/selftest/cases/imagefeatures.py
+++ b/meta/lib/oeqa/selftest/cases/imagefeatures.py
@@ -302,6 +302,165 @@ SKIP_RECIPE[busybox] = "Don't build this"
result = runCmd('objdump --syms %s | grep debug' % t)
self.assertTrue("debug" in result.output, msg='Failed to find debug symbol: %s' % result.output)
+ def test_image_gen_pkgdbfs_ipk(self):
+ """
+ Summary: Check pkgdb snapshot
+ Expected: 1. core-image-minimal can be build with IMAGE_GEN_PKGDBFS variable set
+ 2. *rootfs-pkgdb.tar.gz snapshot of the package manager state is created,
+ after the rootfs of the core-image-minimal is put together
+ 3. check that the restored package index is usable by the default packagemanager
+ PACKAGE_CLASS=package_ipk -> opkg in this case.
+ """
+
+ image = 'core-image-minimal'
+ features = 'IMAGE_GEN_PKGDBFS = "1"\n'
+ features += 'PACKAGE_CLASSES = "package_ipk"\n'
+ self.write_config(features)
+
+ # forcing the regeneration of the rootfs to have an up-to-date package index
+ # then creating the backup = pkgdbfs
+ # and finally forcing the deploy step, to have the pkgdb.tar.gz available in the DEPLOY_DIR_IMAGE
+ bitbake(image + ' -c rootfs -f -c image_pkgdbfs_tar -f -c image_complete -f')
+
+ img_vars = get_bb_vars(['DEPLOY_DIR_IMAGE', 'IMAGE_LINK_NAME', 'MACHINE', 'DEFAULTTUNE'], image)
+
+ tar_file = os.path.join(img_vars['DEPLOY_DIR_IMAGE'], "%s-pkgdb.tar.gz" % (img_vars['IMAGE_LINK_NAME']))
+ self.assertTrue(os.path.exists(tar_file), 'pkgdb filesystem not generated at %s' % tar_file)
+ result = runCmd('cd %s; tar xvf %s' % (img_vars['DEPLOY_DIR_IMAGE'], tar_file))
+ self.assertEqual(result.status, 0, msg='Failed to extract %s: %s' % (tar_file, result.output))
+ self.assertTrue(os.path.exists(os.path.join(img_vars['DEPLOY_DIR_IMAGE'], 'var/lib/opkg/status')), 'opkg\'s status file was not present in: %s' % tar_file)
+
+ bitbake("opkg-native")
+ bitbake("build-sysroots -c build_native_sysroot")
+
+ native_vars = get_bb_vars(["STAGING_DIR", "BUILD_ARCH", "bindir_native", "libdir_native"])
+ stagingbindir = os.path.join(native_vars["STAGING_DIR"], native_vars["BUILD_ARCH"], native_vars["bindir_native"].lstrip(os.sep))
+
+ # some 'add-arch' have '-' replaced by '_', some not; adding both does not hurt
+ cmd = os.path.join(stagingbindir, 'opkg') + \
+ ' --volatile-cache --offline-root=%s --add-arch %s:11 --add-arch %s:12 list-installed' \
+ % (img_vars['DEPLOY_DIR_IMAGE'], img_vars["MACHINE"], img_vars["MACHINE"].replace('-','_'))
+ self.logger.debug('running cmd: %s' % cmd)
+ result = runCmd(cmd)
+ self.logger.debug("list-installed:\n%s" % result.output)
+ self.assertEqual(result.status, 0, msg='Failed to run package manager on unpacked pkgdb %s: %s' % (tar_file, result.output))
+ # check for a 'random, very architecture specific' package
+ if 'qemux86' in img_vars['MACHINE']:
+ self.assertTrue("kernel-image" in result.output, msg='Failed query installed packages')
+ if 'qemuarm' in img_vars['MACHINE']:
+ self.assertTrue("core-boot" in result.output, msg='Failed query installed packages')
+
+ cmd = os.path.join(stagingbindir, 'opkg') + \
+ ' --volatile-cache --offline-root=%s --add-arch %s:11 --add-arch %s:12 list-installed' \
+ % (img_vars['DEPLOY_DIR_IMAGE'], img_vars["DEFAULTTUNE"], img_vars["DEFAULTTUNE"].replace('-','_'))
+ self.logger.debug('running cmd: %s' % cmd)
+ result = runCmd(cmd)
+ self.logger.debug("list-installed:\n%s" % result.output)
+ self.assertEqual(result.status, 0, msg='Failed to run package manager on unpacked pkgdb %s: %s' % (tar_file, result.output))
+ # check for a common base package, a random library for example:
+ self.assertTrue("libc6" in result.output, msg='Failed query installed packages')
+
+ def test_image_gen_pkgdbfs_deb(self):
+ """
+ Summary: Check pkgdb snapshot
+ Expected: 1. core-image-minimal can be build with IMAGE_GEN_PKGDBFS variable set
+ 2. *rootfs-pkgdb.tar.gz snapshot of the package manager state is created,
+ after the rootfs of the core-image-minimal is put together
+ 3. check that the restored package index is usable by the packagemanager
+ PACKAGE_CLASS=package_deb -> dpkg in this case.
+ """
+
+ image = 'core-image-minimal'
+ features = 'IMAGE_GEN_PKGDBFS = "1"\n'
+ features += 'PACKAGE_CLASSES = "package_deb"\n'
+ self.write_config(features)
+
+ # forcing the regeneration of the rootfs to have an up-to-date package index
+ # then creating the backup = pkgdbfs
+ # and finally forcing the deploy step, to have the pkgdb.tar.gz available in the DEPLOY_DIR_IMAGE
+ bitbake(image + ' -c rootfs -f -c image_pkgdbfs_tar -f -c image_complete -f')
+
+ img_vars = get_bb_vars(['DEPLOY_DIR_IMAGE', 'IMAGE_LINK_NAME', 'MACHINE', 'DEFAULTTUNE'], image)
+
+ tar_file = os.path.join(img_vars['DEPLOY_DIR_IMAGE'], "%s-pkgdb.tar.gz" % (img_vars['IMAGE_LINK_NAME']))
+ self.assertTrue(os.path.exists(tar_file), 'pkgdb filesystem not generated at %s' % tar_file)
+ result = runCmd('cd %s; tar xvf %s' % (img_vars['DEPLOY_DIR_IMAGE'], tar_file))
+ self.assertEqual(result.status, 0, msg='Failed to extract %s: %s' % (tar_file, result.output))
+ self.assertTrue(os.path.exists(os.path.join(img_vars['DEPLOY_DIR_IMAGE'], 'var/lib/dpkg/status')), 'dpkg\'s status file was not present in: %s' % tar_file)
+
+ bitbake("dpkg-native")
+ bitbake("build-sysroots -c build_native_sysroot")
+
+ native_vars = get_bb_vars(["STAGING_DIR", "BUILD_ARCH", "bindir_native", "libdir_native"])
+ stagingbindir = os.path.join(native_vars["STAGING_DIR"], native_vars["BUILD_ARCH"], native_vars["bindir_native"].lstrip(os.sep))
+
+ cmd = os.path.join(stagingbindir, 'dpkg') + \
+ ' --admindir=%s/var/lib/dpkg --get-selections' \
+ % (img_vars['DEPLOY_DIR_IMAGE'])
+ self.logger.debug('running cmd: %s' % cmd)
+ result = runCmd(cmd)
+ self.logger.debug("list-installed:\n%s" % result.output)
+ self.assertEqual(result.status, 0, msg='Failed to run package manager on unpacked pkgdb %s: %s' % (tar_file, result.output))
+
+ # check for a 'random, very architecture specific' package
+ if 'qemux86' in img_vars['MACHINE']:
+ self.assertTrue("kernel-image" in result.output, msg='Failed query installed packages')
+ if 'qemuarm' in img_vars['MACHINE']:
+ self.assertTrue("core-boot" in result.output, msg='Failed query installed packages')
+ # check for a common base package, a random library for example:
+ self.assertTrue("libc6" in result.output, msg='Failed query installed packages')
+
+ def test_image_gen_pkgdbfs_rpm(self):
+ """
+ Summary: Check pkgdb snapshot
+ Expected: 1. core-image-minimal can be build with IMAGE_GEN_PKGDBFS variable set
+ 2. *rootfs-pkgdb.tar.gz snapshot of the package manager state is created,
+ after the rootfs of the core-image-minimal is put together
+ 3. check that the restored package index is usable by the packagemanager
+ PACKAGE_CLASS=package_rpm -> dnf in this case.
+ """
+
+ image = 'core-image-minimal'
+ features = 'IMAGE_GEN_PKGDBFS = "1"\n'
+ features += 'PACKAGE_CLASSES = "package_rpm"\n'
+ self.write_config(features)
+
+ # forcing the regeneration of the rootfs to have an up-to-date package index
+ # then creating the backup = pkgdbfs
+ # and finally forcing the deploy step, to have the pkgdb.tar.gz available in the DEPLOY_DIR_IMAGE
+ bitbake(image + ' -c rootfs -f -c image_pkgdbfs_tar -f -c image_complete -f')
+
+ img_vars = get_bb_vars(['DEPLOY_DIR_IMAGE', 'IMAGE_LINK_NAME', 'MACHINE', 'DEFAULTTUNE'], image)
+
+ tar_file = os.path.join(img_vars['DEPLOY_DIR_IMAGE'], "%s-pkgdb.tar.gz" % (img_vars['IMAGE_LINK_NAME']))
+ self.assertTrue(os.path.exists(tar_file), 'pkgdb filesystem not generated at %s' % tar_file)
+ result = runCmd('cd %s; tar xvf %s' % (img_vars['DEPLOY_DIR_IMAGE'], tar_file))
+ self.assertEqual(result.status, 0, msg='Failed to extract %s: %s' % (tar_file, result.output))
+ self.assertTrue(os.path.exists(os.path.join(img_vars['DEPLOY_DIR_IMAGE'], 'var/lib/rpm/rpmdb.sqlite')), 'rpms\'s database file was not present in: %s' % tar_file)
+
+ bitbake("rpm-native")
+ bitbake("build-sysroots -c build_native_sysroot")
+
+ native_vars = get_bb_vars(["STAGING_DIR", "BUILD_ARCH", "bindir_native", "libdir_native"])
+ stagingbindir = os.path.join(native_vars["STAGING_DIR"], native_vars["BUILD_ARCH"], native_vars["bindir_native"].lstrip(os.sep))
+
+ # some 'add-arch' have '-' replaced by '_', some not; adding both does not hurt
+ cmd = os.path.join(stagingbindir, 'rpm') + \
+ ' --dbpath=%s/var/lib/rpm/ --query --all' \
+ % (img_vars['DEPLOY_DIR_IMAGE'])
+ self.logger.debug('running cmd: %s' % cmd)
+ result = runCmd(cmd)
+ self.logger.debug("list-installed:\n%s" % result.output)
+ self.assertEqual(result.status, 0, msg='Failed to run package manager on unpacked pkgdb %s: %s' % (tar_file, result.output))
+
+ # check for a 'random, very architecture specific' package
+ if 'qemux86' in img_vars['MACHINE']:
+ self.assertTrue("kernel-image" in result.output, msg='Failed query installed packages')
+ if 'qemuarm' in img_vars['MACHINE']:
+ self.assertTrue("core-boot" in result.output, msg='Failed query installed packages')
+ # check for a common base package, a random library for example:
+ self.assertTrue("libc6" in result.output, msg='Failed query installed packages')
+
def test_empty_image(self):
"""Test creation of image with no packages"""
image = 'test-empty-image'
--
2.34.1
^ permalink raw reply related [flat|nested] 8+ messages in thread
* [PATCH v9 2/3] image.bbclass/rootfs: set and unpack package-database
2024-05-15 22:34 [PATCH v9 0/3] pkg-database and systemd-sysext image Johannes Schneider
2024-05-15 22:34 ` [PATCH v9 1/3] image.bbclass/rootfs: archive and deploy package database Johannes Schneider
@ 2024-05-15 22:34 ` Johannes Schneider
2024-05-15 22:35 ` [PATCH v9 3/3] classes: add a systemd-sysext image class Johannes Schneider
2024-05-17 18:33 ` [OE-core] [PATCH v9 0/3] pkg-database and systemd-sysext image Alexandre Belloni
3 siblings, 0 replies; 8+ messages in thread
From: Johannes Schneider @ 2024-05-15 22:34 UTC (permalink / raw
To: openembedded-core, richard.purdie, alex.kanavin,
alexandre.belloni
Cc: Johannes Schneider
set the package-database of a "lower image" to unpack and build upon
when installing packages for the current image. This way a lean image
will be created, which only holds the packages that are not already
present in the lower image.
An image build such could then be used with overlayfs or systemd-
sysext to extend the "lower image" on demand; for development purposes
on a device running the "lower image" in RO mode for example.
A configuration could look as follows:
some-core-image.bb
inherit image
IMAGE_GEN_PKGDBFS = "1"
extending-image.bb
inherit image
IMAGE_BASE_PKGDB = "some-core-image"
Signed-off-by: Johannes Schneider <johannes.schneider@leica-geosystems.com>
---
meta/classes-recipe/image.bbclass | 23 ++++++--
meta/conf/documentation.conf | 3 +-
meta/lib/oe/package_manager/deb/rootfs.py | 2 +
meta/lib/oe/package_manager/ipk/rootfs.py | 6 ++-
meta/lib/oe/package_manager/rpm/rootfs.py | 7 ++-
meta/lib/oe/rootfs.py | 18 +++++++
meta/lib/oeqa/selftest/cases/imagefeatures.py | 52 ++++++++++++++++++-
7 files changed, 102 insertions(+), 9 deletions(-)
diff --git a/meta/classes-recipe/image.bbclass b/meta/classes-recipe/image.bbclass
index 3ccaaa17b8..c573c37cd8 100644
--- a/meta/classes-recipe/image.bbclass
+++ b/meta/classes-recipe/image.bbclass
@@ -42,8 +42,16 @@ IMAGE_FEATURES ?= ""
IMAGE_FEATURES[type] = "list"
IMAGE_FEATURES[validitems] += "debug-tweaks read-only-rootfs read-only-rootfs-delayed-postinsts stateless-rootfs empty-root-password allow-empty-password allow-root-login serial-autologin-root post-install-logging overlayfs-etc"
-# Generate snapshot of the package database?
+# Image layering:
+# a "base image" would create a snapshot of the package-database after the
+# installation of all packages into the rootfs is done. The next/other image
+# "layered on-top" of the former would then import that database and install
+# further packages; without reinstalling packages/dependencies that are already
+# installed in the layer below.
+# Set to '1' in a "base image" recipe, to preserve a snapshot of the package database.
IMAGE_GEN_PKGDBFS ?= "0"
+# "PN" of a "base image", upon which the current image is to be built upon.
+IMAGE_BASE_PKGDB ?= ""
# Generate companion debugfs?
IMAGE_GEN_DEBUGFS ?= "0"
@@ -118,6 +126,15 @@ do_rootfs[depends] += " \
"
do_rootfs[recrdeptask] += "do_packagedata"
+python () {
+ # make sure that the 'base image' has been queued in before this
+ # image wants to unpack and build upon the formers pgkdb
+ base_image = d.getVar('IMAGE_BASE_PKGDB')
+ pn = d.getVar('PN')
+ if base_image and base_image != pn:
+ d.appendVarFlag("do_rootfs", 'depends', ' '+ base_image + ':do_image_complete')
+}
+
def rootfs_command_variables(d):
return ['ROOTFS_POSTPROCESS_COMMAND','ROOTFS_PREPROCESS_COMMAND','ROOTFS_POSTINSTALL_COMMAND','ROOTFS_POSTUNINSTALL_COMMAND','OPKG_PREPROCESS_COMMANDS','OPKG_POSTPROCESS_COMMANDS','IMAGE_POSTPROCESS_COMMAND',
'IMAGE_PREPROCESS_COMMAND','RPM_PREPROCESS_COMMANDS','RPM_POSTPROCESS_COMMANDS','DEB_PREPROCESS_COMMANDS','DEB_POSTPROCESS_COMMANDS']
@@ -134,8 +151,8 @@ def rootfs_variables(d):
'IMAGE_ROOTFS_MAXSIZE','IMAGE_NAME','IMAGE_LINK_NAME','IMAGE_MANIFEST','DEPLOY_DIR_IMAGE','IMAGE_FSTYPES','IMAGE_INSTALL_COMPLEMENTARY','IMAGE_LINGUAS', 'IMAGE_LINGUAS_COMPLEMENTARY', 'IMAGE_LOCALES_ARCHIVE',
'MULTILIBRE_ALLOW_REP','MULTILIB_TEMP_ROOTFS','MULTILIB_VARIANTS','MULTILIBS','ALL_MULTILIB_PACKAGE_ARCHS','MULTILIB_GLOBAL_VARIANTS','BAD_RECOMMENDATIONS','NO_RECOMMENDATIONS',
'PACKAGE_ARCHS','PACKAGE_CLASSES','TARGET_VENDOR','TARGET_ARCH','TARGET_OS','OVERRIDES','BBEXTENDVARIANT','FEED_DEPLOYDIR_BASE_URI','INTERCEPT_DIR','USE_DEVFS',
- 'CONVERSIONTYPES', 'IMAGE_GEN_PKGDBFS', 'IMAGE_GEN_DEBUGFS', 'ROOTFS_RO_UNNEEDED', 'IMGDEPLOYDIR', 'PACKAGE_EXCLUDE_COMPLEMENTARY', 'REPRODUCIBLE_TIMESTAMP_ROOTFS',
- 'IMAGE_INSTALL_DEBUGFS']
+ 'CONVERSIONTYPES', 'IMAGE_GEN_PKGDBFS', 'IMAGE_BASE_PKGDB', 'IMAGE_GEN_DEBUGFS', 'ROOTFS_RO_UNNEEDED', 'IMGDEPLOYDIR', 'PACKAGE_EXCLUDE_COMPLEMENTARY',
+ 'REPRODUCIBLE_TIMESTAMP_ROOTFS', 'IMAGE_INSTALL_DEBUGFS']
variables.extend(rootfs_command_variables(d))
variables.extend(variable_depends(d))
return " ".join(variables)
diff --git a/meta/conf/documentation.conf b/meta/conf/documentation.conf
index 36aebb59ab..9f493cfe96 100644
--- a/meta/conf/documentation.conf
+++ b/meta/conf/documentation.conf
@@ -208,6 +208,7 @@ ICECC_PATH[doc] = "The location of the icecc binary."
ICECC_CLASS_DISABLE[doc] = "Identifies user classes that you do not want the Icecream distributed compile support to consider."
ICECC_RECIPE_DISABLE[doc] = "Identifies user recipes that you do not want the Icecream distributed compile support to consider."
ICECC_RECIPE_ENABLE[doc] = "Identifies user recipes that use an empty PARALLEL_MAKE variable that you want to force remote distributed compilation on using the Icecream distributed compile support."
+IMAGE_BASE_PKGDB[doc] = "Set to the PN of a base-image name, which was built with IMAGE_GEN_PKGDBFS enabled; Then this image will only add/install packages into its rootfs that are not already in the base-image."
IMAGE_BASENAME[doc] = "The base name of image output files."
IMAGE_BOOT_FILES[doc] = "Whitespace separated list of files from ${DEPLOY_DIR_IMAGE} to place in boot partition. Entries will be installed under a same name as the source file. To change the destination file name, pass a desired name after a semicolon (eg. u-boot.img;uboot)."
IMAGE_CLASSES[doc] = "A list of classes that all images should inherit."
@@ -215,7 +216,7 @@ IMAGE_FEATURES[doc] = "The primary list of features to include in an image. Conf
IMAGE_FSTYPES[doc] = "Formats of root filesystem images that you want to have created."
IMAGE_FSTYPES_DEBUGFS[doc] = "Formats of the debug root filesystem images that you want to have created."
IMAGE_GEN_DEBUGFS[doc] = "When set to '1', generate a companion debug object/source filesystem image."
-IMAGE_GEN_PKGDBFS[doc] = "When set to '1', create a snapshot of the package-manager state after the rootfs has been assembled."
+IMAGE_GEN_PKGDBFS[doc] = "When set to '1', create a snapshot of the package-manager state after the rootfs has been assembled; This snapshot can then be used in a different image recipe, that sets IMAGE_BASE_PKGDB to this images PN."
IMAGE_INSTALL[doc] = "Specifies the packages to install into an image. Image recipes set IMAGE_INSTALL to specify the packages to install into an image through image.bbclass."
IMAGE_LINGUAS[doc] = "Specifies the list of locales to install into the image during the root filesystem construction process."
IMAGE_NAME[doc] = "The name of the output image files minus the extension."
diff --git a/meta/lib/oe/package_manager/deb/rootfs.py b/meta/lib/oe/package_manager/deb/rootfs.py
index 43107c8663..71a21df09b 100644
--- a/meta/lib/oe/package_manager/deb/rootfs.py
+++ b/meta/lib/oe/package_manager/deb/rootfs.py
@@ -152,6 +152,8 @@ class PkgRootfs(DpkgOpkgRootfs):
execute_pre_post_process(self.d, deb_pre_process_cmds)
+ self._unpack_pkg_db_rootfs(['/var/lib/dpkg'])
+
if self.progress_reporter:
self.progress_reporter.next_stage()
# Don't support incremental, so skip that
diff --git a/meta/lib/oe/package_manager/ipk/rootfs.py b/meta/lib/oe/package_manager/ipk/rootfs.py
index 64d9bc7969..408faa8030 100644
--- a/meta/lib/oe/package_manager/ipk/rootfs.py
+++ b/meta/lib/oe/package_manager/ipk/rootfs.py
@@ -276,12 +276,16 @@ class PkgRootfs(DpkgOpkgRootfs):
pkgs_to_install = self.manifest.parse_initial_manifest()
opkg_pre_process_cmds = self.d.getVar('OPKG_PREPROCESS_COMMANDS')
opkg_post_process_cmds = self.d.getVar('OPKG_POSTPROCESS_COMMANDS')
+ opkg_lib_dir = self.d.getVar('OPKGLIBDIR')
+ opkg_dir = os.path.join(opkg_lib_dir, 'opkg')
# update PM index files
self.pm.write_index()
execute_pre_post_process(self.d, opkg_pre_process_cmds)
+ self._unpack_pkg_db_rootfs([opkg_dir])
+
if self.progress_reporter:
self.progress_reporter.next_stage()
# Steps are a bit different in order, skip next
@@ -317,8 +321,6 @@ class PkgRootfs(DpkgOpkgRootfs):
if self.progress_reporter:
self.progress_reporter.next_stage()
- opkg_lib_dir = self.d.getVar('OPKGLIBDIR')
- opkg_dir = os.path.join(opkg_lib_dir, 'opkg')
self._setup_pkg_db_rootfs([opkg_dir])
self._setup_dbg_rootfs([opkg_dir])
diff --git a/meta/lib/oe/package_manager/rpm/rootfs.py b/meta/lib/oe/package_manager/rpm/rootfs.py
index 673006c131..9986b13b5f 100644
--- a/meta/lib/oe/package_manager/rpm/rootfs.py
+++ b/meta/lib/oe/package_manager/rpm/rootfs.py
@@ -67,12 +67,15 @@ class PkgRootfs(Rootfs):
pkgs_to_install = self.manifest.parse_initial_manifest()
rpm_pre_process_cmds = self.d.getVar('RPM_PREPROCESS_COMMANDS')
rpm_post_process_cmds = self.d.getVar('RPM_POSTPROCESS_COMMANDS')
+ package_paths = ['/etc/rpm', '/etc/rpmrc', '/etc/dnf', '/var/lib/rpm', '/var/cache/dnf', '/var/lib/dnf']
# update PM index files
self.pm.write_index()
execute_pre_post_process(self.d, rpm_pre_process_cmds)
+ self._unpack_pkg_db_rootfs(package_paths)
+
if self.progress_reporter:
self.progress_reporter.next_stage()
@@ -110,8 +113,8 @@ class PkgRootfs(Rootfs):
if self.progress_reporter:
self.progress_reporter.next_stage()
- self._setup_pkg_db_rootfs(['/etc/rpm', '/etc/rpmrc', '/etc/dnf', '/var/lib/rpm', '/var/cache/dnf', '/var/lib/dnf'])
- self._setup_dbg_rootfs(['/etc/rpm', '/etc/rpmrc', '/etc/dnf', '/var/lib/rpm', '/var/cache/dnf', '/var/lib/dnf'])
+ self._setup_pkg_db_rootfs(package_paths)
+ self._setup_dbg_rootfs(package_paths)
execute_pre_post_process(self.d, rpm_post_process_cmds)
diff --git a/meta/lib/oe/rootfs.py b/meta/lib/oe/rootfs.py
index 6d6e888a30..9d691c58cd 100644
--- a/meta/lib/oe/rootfs.py
+++ b/meta/lib/oe/rootfs.py
@@ -106,6 +106,24 @@ class Rootfs(object, metaclass=ABCMeta):
def _cleanup(self):
pass
+ def _unpack_pkg_db_rootfs(self, package_paths):
+ import tarfile
+ pn = self.d.getVar('PN')
+ base_pkgdb = self.d.getVar('IMAGE_BASE_PKGDB')
+ if not base_pkgdb or pn == base_pkgdb:
+ return
+
+ fname = self.d.getVar('DEPLOY_DIR_IMAGE') + '/' + self.d.getVar('IMAGE_BASE_PKGDB') + '-' + self.d.getVar('MACHINE') + '.rootfs-pkgdb.tar.gz'
+ if not os.path.exists(fname):
+ bb.fatal("PKGDB for '%s' does not exist,\n\texpected: %s\n\tthe recipe for '%s' should set: IMAGE_GEN_PKGDBFS='1'" % (base_pkgdb, fname, base_pkgdb))
+ return
+
+ bb.note(" unpacking base image package database...")
+ if fname.endswith("tar.gz"):
+ tar = tarfile.open(fname, "r:gz")
+ tar.extractall(path=self.image_rootfs)
+ tar.close()
+
def _setup_pkg_db_rootfs(self, package_paths):
gen_pkg_db_fs = bb.utils.to_boolean(self.d.getVar('IMAGE_GEN_PKGDBFS'))
if not gen_pkg_db_fs:
diff --git a/meta/lib/oeqa/selftest/cases/imagefeatures.py b/meta/lib/oeqa/selftest/cases/imagefeatures.py
index 6724bac0e4..560bb0747f 100644
--- a/meta/lib/oeqa/selftest/cases/imagefeatures.py
+++ b/meta/lib/oeqa/selftest/cases/imagefeatures.py
@@ -429,7 +429,6 @@ SKIP_RECIPE[busybox] = "Don't build this"
# then creating the backup = pkgdbfs
# and finally forcing the deploy step, to have the pkgdb.tar.gz available in the DEPLOY_DIR_IMAGE
bitbake(image + ' -c rootfs -f -c image_pkgdbfs_tar -f -c image_complete -f')
-
img_vars = get_bb_vars(['DEPLOY_DIR_IMAGE', 'IMAGE_LINK_NAME', 'MACHINE', 'DEFAULTTUNE'], image)
tar_file = os.path.join(img_vars['DEPLOY_DIR_IMAGE'], "%s-pkgdb.tar.gz" % (img_vars['IMAGE_LINK_NAME']))
@@ -461,6 +460,57 @@ SKIP_RECIPE[busybox] = "Don't build this"
# check for a common base package, a random library for example:
self.assertTrue("libc6" in result.output, msg='Failed query installed packages')
+ def test_image_use_pkgdbfs(self):
+ """
+ Summary: Check pkgdb snapshot
+ Expected: 1. core-image-minimal is built with IMAGE_GEN_PKGDBFS variable set
+ 2. core-image-minimal-mtdutils is built with:
+ IMAGE_BASE_PKGDB pointing to the former, to build upon it
+ 3. the rootfs of both images is checked, and there should be no file
+ from any of the installed packages that shows up in both images
+ Note: 'using' the packagemanager database is actually packagemanager agnostic;
+ as long as the 'gen' took a usable backup of the managers database/state.
+ """
+
+ base_image = 'core-image-minimal'
+ extension_image = 'core-image-minimal-mtdutils'
+
+ features = 'IMAGE_FSTYPES = "tar.bz2"\n'
+ # enable usrmerge so that all relevant pieces are in one place = /usr
+ features += 'DISTRO_FEATURES += "usrmerge"\n'
+ features += 'IMAGE_GEN_PKGDBFS = "1"\n'
+ features += 'IMAGE_BASE_PKGDB = "%s"\n' % base_image
+ self.write_config(features)
+
+ # forcing the regeneration of the rootfs to have an up-to-date package index
+ # then creating the backup = pkgdbfs
+ # and finally forcing the deploy step, to have the pkgdb.tar.gz available in the DEPLOY_DIR_IMAGE
+ bitbake(base_image + ' -c rootfs -f -c image_pkgdbfs_tar -f -c image_complete -f')
+ bitbake(extension_image)
+
+ # collect image content of both images
+ img_vars = get_bb_vars(['DEPLOY_DIR_IMAGE', 'IMAGE_LINK_NAME'], base_image)
+ img_file = os.path.join(img_vars['DEPLOY_DIR_IMAGE'], "%s.%s" % (img_vars['IMAGE_LINK_NAME'], 'tar.bz2'))
+ self.logger.debug("base image: %s" % img_file)
+ img_content = runCmd('tar -tf %s' % img_file)
+
+ ext_vars = get_bb_vars(['DEPLOY_DIR_IMAGE', 'IMAGE_LINK_NAME'], extension_image)
+ ext_file = os.path.join(ext_vars['DEPLOY_DIR_IMAGE'], "%s.%s" % (ext_vars['IMAGE_LINK_NAME'], 'tar.bz2'))
+ self.logger.debug("extension image: %s" % ext_file)
+ ext_content = runCmd('tar -tf %s' % ext_file)
+
+ # filter out folders and anything outside of /usr
+ img_files = [ x for x in img_content.output.splitlines() if not x.endswith('/') and x.startswith('./usr/') ]
+ self.logger.debug("image files:\n%s" % img_files)
+ ext_files = [ x for x in ext_content.output.splitlines() if not x.endswith('/') and x.startswith('./usr/') ]
+
+ # check mutual exclusive content listing
+ self.logger.debug("extension files:\n%s" % ext_files)
+ for i in img_files:
+ self.assertTrue(i not in ext_files, "found %s in extension" % i)
+ for e in ext_files:
+ self.assertTrue(e not in img_files, "found %s in base-image" % e)
+
def test_empty_image(self):
"""Test creation of image with no packages"""
image = 'test-empty-image'
--
2.34.1
^ permalink raw reply related [flat|nested] 8+ messages in thread
* [PATCH v9 3/3] classes: add a systemd-sysext image class
2024-05-15 22:34 [PATCH v9 0/3] pkg-database and systemd-sysext image Johannes Schneider
2024-05-15 22:34 ` [PATCH v9 1/3] image.bbclass/rootfs: archive and deploy package database Johannes Schneider
2024-05-15 22:34 ` [PATCH v9 2/3] image.bbclass/rootfs: set and unpack package-database Johannes Schneider
@ 2024-05-15 22:35 ` Johannes Schneider
2024-05-17 18:33 ` [OE-core] [PATCH v9 0/3] pkg-database and systemd-sysext image Alexandre Belloni
3 siblings, 0 replies; 8+ messages in thread
From: Johannes Schneider @ 2024-05-15 22:35 UTC (permalink / raw
To: openembedded-core, richard.purdie, alex.kanavin,
alexandre.belloni
Cc: Johannes Schneider
systemd-sysext can load a raw-image containing usr/ and opt/ folders
to mount them as RO overlay over the rootfs, to "extend" the systems.
This class provides the necessary changes/additions to the enclosed
file-system so that systemd-sysext accepts the extension for "merge"
into the rootfs.
With such an created image, placed into the correct folder (see [1]),
`systemd-sysext list` should be able to list the "extension" and
`systemd-sysext merge` should enable the overlay. On both commands a
preceding "SYSTEMD_LOG_LEVEL=debug" can aide in figuring out what is
amiss.
The strict name checking systemd-sysext does against the name of
extension-release.NAME file, is disabled, as there is only one such in
the resulting image. This is done to allow a user to freely rename the
resulting image file.
Note that for e.g. squashfs, the kernel needs CONFIG_SQUASHFS_XATTR=y
Link: https://www.freedesktop.org/software/systemd/man/latest/systemd-sysext.html
Link: https://0pointer.net/blog/testing-my-system-code-in-usr-without-modifying-usr.html
Signed-off-by: Johannes Schneider <johannes.schneider@leica-geosystems.com>
---
meta/classes-recipe/image-sysext.bbclass | 43 ++++++++++++++++++++++++
1 file changed, 43 insertions(+)
create mode 100644 meta/classes-recipe/image-sysext.bbclass
diff --git a/meta/classes-recipe/image-sysext.bbclass b/meta/classes-recipe/image-sysext.bbclass
new file mode 100644
index 0000000000..bc3e4d52b5
--- /dev/null
+++ b/meta/classes-recipe/image-sysext.bbclass
@@ -0,0 +1,43 @@
+# SPDX-License-Identifier: MIT
+#
+# Copyright Leica Geosystems AG
+#
+
+# systemd-sysext [1] has a simple mechanism for version compatibility:
+# the extension to be loaded has to contain a
+# /usr/lib/extension-release.d/extension-release.NAME
+# with "NAME" *exactly* matching the filename of the extensions
+# raw-device filename/
+#
+# from the extension-release file the "ID" and "VERSION_ID" fields are
+# matched against the etc/os-release and the extension is only "merged"
+# if no mismatches between NAME, ID, and VERSION_ID.
+#
+# Link: https://www.freedesktop.org/software/systemd/man/latest/systemd-sysext.html
+
+inherit image
+
+IMAGE_NAME_SUFFIX = ".sysext"
+EXTENSION_NAME = "${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${IMAGE_FSTYPES}"
+IMAGE_LINK_NAME:append = ".sysext"
+
+DEPENDS += " os-release"
+
+sysext_image_mangle_rootfs() {
+ R=${IMAGE_ROOTFS}
+
+ # pull a copy of the rootfs version information, which systemd-sysext matches against
+ cp -av ${RECIPE_SYSROOT}/${nonarch_libdir}/os-release ${WORKDIR}/extension-release.base
+
+ echo 'EXTENSION_RELOAD_MANAGER=1' >> ${WORKDIR}/extension-release.base
+
+ install -d $R${nonarch_libdir}/extension-release.d
+ install -m 0644 ${WORKDIR}/extension-release.base \
+ $R${nonarch_libdir}/extension-release.d/extension-release.${EXTENSION_NAME}
+
+ # disable systemd-sysext's strict name checking, so that the image file can be renamed, while still being 'merge'-able
+ setfattr -n user.extension-release.strict -v false \
+ $R${nonarch_libdir}/extension-release.d/extension-release.${EXTENSION_NAME}
+}
+
+ROOTFS_POSTPROCESS_COMMAND += " sysext_image_mangle_rootfs; "
--
2.34.1
^ permalink raw reply related [flat|nested] 8+ messages in thread
* Re: [OE-core] [PATCH v9 0/3] pkg-database and systemd-sysext image
2024-05-15 22:34 [PATCH v9 0/3] pkg-database and systemd-sysext image Johannes Schneider
` (2 preceding siblings ...)
2024-05-15 22:35 ` [PATCH v9 3/3] classes: add a systemd-sysext image class Johannes Schneider
@ 2024-05-17 18:33 ` Alexandre Belloni
2024-05-18 6:30 ` SCHNEIDER Johannes
3 siblings, 1 reply; 8+ messages in thread
From: Alexandre Belloni @ 2024-05-17 18:33 UTC (permalink / raw
To: johannes.schneider; +Cc: openembedded-core, richard.purdie, alex.kanavin
Hello,
deb and rpm seem successful but ipk still fails:
https://autobuilder.yoctoproject.org/typhoon/#/builders/127/builds/3335/steps/14/logs/stdio
On 16/05/2024 00:34:57+0200, Johannes Schneider via lists.openembedded.org wrote:
> systemd-sysext allows to overlay another image (or multiple) ontop of
> a "base-image" = the current rootfs, via the use of overlayfs; to add
> tools and features meant for development purposes.
>
> To quote the documentation on systemd-sysext:
> " ...addition in order to make debugging/development easier). System
> extension images should not be misunderstood as a generic software
> packaging framework, ..."
>
> To build a lean image, that only holds packages that are not already
> part of the base-image, a snapshot of the package-database is taken
> after the installation of the base-rootfs is done, and picked up again
> when collecting the rootfs of such a extension image.
>
> with all this in place an example usage could look like this:
> some-core-image.bb
> inherit core-image
> IMAGE_GEN_PKGDBFS = "1"
>
> extending-image.bb
> inherit image-sysext
> IMAGE_FSTYPES = "squashfs"
> IMAGE_BASE_PKGDB = "some-core-image"
> # the above pointing at a package-db similar to:
> # build/deploy/images/$MACHINE/some-core-image-$MACHINE-20240210172305-pkgdb.rootfs.tar.gz
>
> then on the device, running some-core-image, with the extension image placed at FN:
> $> ln -s "$FN" /run/extensions/$(basename $FN).raw
> $> systemd-sysext list
> $> SYSTEMD_LOG_LEVEL=debug systemd-sysext merge
>
> As long as the VERSION_ID of the extension image matches the os-release
> in the base image, the above commands return sucessfully;
> for details on the compativility check see the docs for systemd-sysext.
>
> =========
>
> changes with v2:
> rebase from 'kirkstone' onto 'master'
>
> changes with v3:
> incorporate review suggestions for simplification
> add task dependency handling
> add oe-selftest for the pkgdb handling
> add variable documentation and
> some more comments, and examples in the commit-msg
>
> changes with v4:
> rebase onto 'master' => no functional changes
> fixed patchtest findings
>
> changes with v5:
> rebase onto 'master'
> add '.sysext' to the deployed symlink name
> sidenote on the tests and autobuilder failure: run locally they succeed, e.g.:
> #> oe-selftest --verbose -r imagefeatures.ImageFeatures.test_image_gen_pkgdbfs
>
> changes with v6:
> tests: restructure to call 'bitbake' only once in the testcase itself
> (in hopes of solving the autobuilder problem; local test runs succeed)
>
> changes with v7:
> tests: undo the restructuring of done in v6, in favour of explicitly forcing steps:
> '-c rootfs -f'. since concurrency of the autobuilder, and reusing artifacts/sstate-caching is probably the issue?
>
> changes with v8:
> tests: fixed missing feature=PACKAGE_CLASSES during 'test_image_gen_pkgdbfs'
> which is now split into three, one for each of the three available packagemanager: ipk, deb, rpm
>
> changes with v9:
> tests: fix singular failure in 'oe-selftest-armhost'
> which was due to the built image not containing a 'kernel-image...', which the gen-pkgdb sanity
> check expected, but 'packagegroup-core-boot' instead
>
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> View/Reply Online (#199430): https://lists.openembedded.org/g/openembedded-core/message/199430
> Mute This Topic: https://lists.openembedded.org/mt/106124365/3617179
> Group Owner: openembedded-core+owner@lists.openembedded.org
> Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [alexandre.belloni@bootlin.com]
> -=-=-=-=-=-=-=-=-=-=-=-
>
--
Alexandre Belloni, co-owner and COO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [OE-core] [PATCH v9 0/3] pkg-database and systemd-sysext image
2024-05-17 18:33 ` [OE-core] [PATCH v9 0/3] pkg-database and systemd-sysext image Alexandre Belloni
@ 2024-05-18 6:30 ` SCHNEIDER Johannes
2024-05-21 10:08 ` Alexander Kanavin
0 siblings, 1 reply; 8+ messages in thread
From: SCHNEIDER Johannes @ 2024-05-18 6:30 UTC (permalink / raw
To: Alexandre Belloni
Cc: openembedded-core@lists.openembedded.org,
richard.purdie@linuxfoundation.org, alex.kanavin@gmail.com
Hoi Alexandre,
is there a way to get more output out of the autobuilder?
the logs only tell "that" the test failed, not "why" = which assertion was missed
if i run the test locally with the differing variables set like this:
#> SDKMACHINE=aarch64 MACHINE=qemuarm64 oe-selftest --verbose -r imagefeatures.ImageFeatures.test_image_gen_pkgdbfs_ipk
it succeeds :-S
... ideas/thoughts?
gruß
Johannes
________________________________________
From: Alexandre Belloni <alexandre.belloni@bootlin.com>
Sent: Friday, May 17, 2024 20:33
To: SCHNEIDER Johannes
Cc: openembedded-core@lists.openembedded.org; richard.purdie@linuxfoundation.org; alex.kanavin@gmail.com
Subject: Re: [OE-core] [PATCH v9 0/3] pkg-database and systemd-sysext image
This email is not from Hexagon’s Office 365 instance. Please be careful while clicking links, opening attachments, or replying to this email.
Hello,
deb and rpm seem successful but ipk still fails:
https://autobuilder.yoctoproject.org/typhoon/#/builders/127/builds/3335/steps/14/logs/stdio
On 16/05/2024 00:34:57+0200, Johannes Schneider via lists.openembedded.org wrote:
> systemd-sysext allows to overlay another image (or multiple) ontop of
> a "base-image" = the current rootfs, via the use of overlayfs; to add
> tools and features meant for development purposes.
>
> To quote the documentation on systemd-sysext:
> " ...addition in order to make debugging/development easier). System
> extension images should not be misunderstood as a generic software
> packaging framework, ..."
>
> To build a lean image, that only holds packages that are not already
> part of the base-image, a snapshot of the package-database is taken
> after the installation of the base-rootfs is done, and picked up again
> when collecting the rootfs of such a extension image.
>
> with all this in place an example usage could look like this:
> some-core-image.bb
> inherit core-image
> IMAGE_GEN_PKGDBFS = "1"
>
> extending-image.bb
> inherit image-sysext
> IMAGE_FSTYPES = "squashfs"
> IMAGE_BASE_PKGDB = "some-core-image"
> # the above pointing at a package-db similar to:
> # build/deploy/images/$MACHINE/some-core-image-$MACHINE-20240210172305-pkgdb.rootfs.tar.gz
>
> then on the device, running some-core-image, with the extension image placed at FN:
> $> ln -s "$FN" /run/extensions/$(basename $FN).raw
> $> systemd-sysext list
> $> SYSTEMD_LOG_LEVEL=debug systemd-sysext merge
>
> As long as the VERSION_ID of the extension image matches the os-release
> in the base image, the above commands return sucessfully;
> for details on the compativility check see the docs for systemd-sysext.
>
> =========
>
> changes with v2:
> rebase from 'kirkstone' onto 'master'
>
> changes with v3:
> incorporate review suggestions for simplification
> add task dependency handling
> add oe-selftest for the pkgdb handling
> add variable documentation and
> some more comments, and examples in the commit-msg
>
> changes with v4:
> rebase onto 'master' => no functional changes
> fixed patchtest findings
>
> changes with v5:
> rebase onto 'master'
> add '.sysext' to the deployed symlink name
> sidenote on the tests and autobuilder failure: run locally they succeed, e.g.:
> #> oe-selftest --verbose -r imagefeatures.ImageFeatures.test_image_gen_pkgdbfs
>
> changes with v6:
> tests: restructure to call 'bitbake' only once in the testcase itself
> (in hopes of solving the autobuilder problem; local test runs succeed)
>
> changes with v7:
> tests: undo the restructuring of done in v6, in favour of explicitly forcing steps:
> '-c rootfs -f'. since concurrency of the autobuilder, and reusing artifacts/sstate-caching is probably the issue?
>
> changes with v8:
> tests: fixed missing feature=PACKAGE_CLASSES during 'test_image_gen_pkgdbfs'
> which is now split into three, one for each of the three available packagemanager: ipk, deb, rpm
>
> changes with v9:
> tests: fix singular failure in 'oe-selftest-armhost'
> which was due to the built image not containing a 'kernel-image...', which the gen-pkgdb sanity
> check expected, but 'packagegroup-core-boot' instead
>
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> View/Reply Online (#199430): https://lists.openembedded.org/g/openembedded-core/message/199430
> Mute This Topic: https://lists.openembedded.org/mt/106124365/3617179
> Group Owner: openembedded-core+owner@lists.openembedded.org
> Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [alexandre.belloni@bootlin.com]
> -=-=-=-=-=-=-=-=-=-=-=-
>
--
Alexandre Belloni, co-owner and COO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com/
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [OE-core] [PATCH v9 0/3] pkg-database and systemd-sysext image
2024-05-18 6:30 ` SCHNEIDER Johannes
@ 2024-05-21 10:08 ` Alexander Kanavin
2024-05-22 10:59 ` SCHNEIDER Johannes
0 siblings, 1 reply; 8+ messages in thread
From: Alexander Kanavin @ 2024-05-21 10:08 UTC (permalink / raw
To: SCHNEIDER Johannes
Cc: Alexandre Belloni, openembedded-core@lists.openembedded.org,
richard.purdie@linuxfoundation.org
If you scroll further up, there's:
2024-05-16 22:05:57,101 - oe-selftest - INFO - FAIL:
imagefeatures.ImageFeatures.test_image_gen_pkgdbfs_ipk
(subunit.RemotedTestCase)
2024-05-16 22:05:57,102 - oe-selftest - INFO -
----------------------------------------------------------------------
2024-05-16 22:05:57,102 - oe-selftest - INFO -
testtools.testresult.real._StringException: Traceback (most recent
call last):
File "/home/pokybuild/yocto-worker/oe-selftest-armhost/build/meta/lib/oeqa/selftest/cases/imagefeatures.py",
line 344, in test_image_gen_pkgdbfs_ipk
result = runCmd(cmd)
File "/home/pokybuild/yocto-worker/oe-selftest-armhost/build/meta/lib/oeqa/utils/commands.py",
line 212, in runCmd
raise AssertionError("Command '%s' returned non-zero exit status
%d:\n%s" % (command, result.status, exc_output))
AssertionError: Command
'/home/pokybuild/yocto-worker/oe-selftest-armhost/build/build-st-4137794/tmp/sysroots/aarch64/usr/bin/opkg
--volatile-cache
--offline-root=/home/pokybuild/yocto-worker/oe-selftest-armhost/build/build-st-4137794/tmp/deploy/images/qemuarm64
--add-arch qemuarm64:11 --add-arch qemuarm64:12 list-installed'
returned non-zero exit status 255:
Also note that this happened only on the arm host, and not on x86
hosts. Which does make it tricky to reproduce, but at least gives a
lead:
https://autobuilder.yoctoproject.org/typhoon/#/builders/83/builds/6921
Alex
On Sat, 18 May 2024 at 08:30, SCHNEIDER Johannes
<johannes.schneider@leica-geosystems.com> wrote:
>
> Hoi Alexandre,
>
>
> is there a way to get more output out of the autobuilder?
> the logs only tell "that" the test failed, not "why" = which assertion was missed
>
> if i run the test locally with the differing variables set like this:
> #> SDKMACHINE=aarch64 MACHINE=qemuarm64 oe-selftest --verbose -r imagefeatures.ImageFeatures.test_image_gen_pkgdbfs_ipk
>
> it succeeds :-S
> ... ideas/thoughts?
>
>
> gruß
> Johannes
>
> ________________________________________
> From: Alexandre Belloni <alexandre.belloni@bootlin.com>
> Sent: Friday, May 17, 2024 20:33
> To: SCHNEIDER Johannes
> Cc: openembedded-core@lists.openembedded.org; richard.purdie@linuxfoundation.org; alex.kanavin@gmail.com
> Subject: Re: [OE-core] [PATCH v9 0/3] pkg-database and systemd-sysext image
>
> This email is not from Hexagon’s Office 365 instance. Please be careful while clicking links, opening attachments, or replying to this email.
>
>
> Hello,
>
> deb and rpm seem successful but ipk still fails:
>
> https://autobuilder.yoctoproject.org/typhoon/#/builders/127/builds/3335/steps/14/logs/stdio
>
> On 16/05/2024 00:34:57+0200, Johannes Schneider via lists.openembedded.org wrote:
> > systemd-sysext allows to overlay another image (or multiple) ontop of
> > a "base-image" = the current rootfs, via the use of overlayfs; to add
> > tools and features meant for development purposes.
> >
> > To quote the documentation on systemd-sysext:
> > " ...addition in order to make debugging/development easier). System
> > extension images should not be misunderstood as a generic software
> > packaging framework, ..."
> >
> > To build a lean image, that only holds packages that are not already
> > part of the base-image, a snapshot of the package-database is taken
> > after the installation of the base-rootfs is done, and picked up again
> > when collecting the rootfs of such a extension image.
> >
> > with all this in place an example usage could look like this:
> > some-core-image.bb
> > inherit core-image
> > IMAGE_GEN_PKGDBFS = "1"
> >
> > extending-image.bb
> > inherit image-sysext
> > IMAGE_FSTYPES = "squashfs"
> > IMAGE_BASE_PKGDB = "some-core-image"
> > # the above pointing at a package-db similar to:
> > # build/deploy/images/$MACHINE/some-core-image-$MACHINE-20240210172305-pkgdb.rootfs.tar.gz
> >
> > then on the device, running some-core-image, with the extension image placed at FN:
> > $> ln -s "$FN" /run/extensions/$(basename $FN).raw
> > $> systemd-sysext list
> > $> SYSTEMD_LOG_LEVEL=debug systemd-sysext merge
> >
> > As long as the VERSION_ID of the extension image matches the os-release
> > in the base image, the above commands return sucessfully;
> > for details on the compativility check see the docs for systemd-sysext.
> >
> > =========
> >
> > changes with v2:
> > rebase from 'kirkstone' onto 'master'
> >
> > changes with v3:
> > incorporate review suggestions for simplification
> > add task dependency handling
> > add oe-selftest for the pkgdb handling
> > add variable documentation and
> > some more comments, and examples in the commit-msg
> >
> > changes with v4:
> > rebase onto 'master' => no functional changes
> > fixed patchtest findings
> >
> > changes with v5:
> > rebase onto 'master'
> > add '.sysext' to the deployed symlink name
> > sidenote on the tests and autobuilder failure: run locally they succeed, e.g.:
> > #> oe-selftest --verbose -r imagefeatures.ImageFeatures.test_image_gen_pkgdbfs
> >
> > changes with v6:
> > tests: restructure to call 'bitbake' only once in the testcase itself
> > (in hopes of solving the autobuilder problem; local test runs succeed)
> >
> > changes with v7:
> > tests: undo the restructuring of done in v6, in favour of explicitly forcing steps:
> > '-c rootfs -f'. since concurrency of the autobuilder, and reusing artifacts/sstate-caching is probably the issue?
> >
> > changes with v8:
> > tests: fixed missing feature=PACKAGE_CLASSES during 'test_image_gen_pkgdbfs'
> > which is now split into three, one for each of the three available packagemanager: ipk, deb, rpm
> >
> > changes with v9:
> > tests: fix singular failure in 'oe-selftest-armhost'
> > which was due to the built image not containing a 'kernel-image...', which the gen-pkgdb sanity
> > check expected, but 'packagegroup-core-boot' instead
>
> >
> > -=-=-=-=-=-=-=-=-=-=-=-
> > Links: You receive all messages sent to this group.
> > View/Reply Online (#199430): https://lists.openembedded.org/g/openembedded-core/message/199430
> > Mute This Topic: https://lists.openembedded.org/mt/106124365/3617179
> > Group Owner: openembedded-core+owner@lists.openembedded.org
> > Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [alexandre.belloni@bootlin.com]
> > -=-=-=-=-=-=-=-=-=-=-=-
> >
>
>
> --
> Alexandre Belloni, co-owner and COO, Bootlin
> Embedded Linux and Kernel engineering
> https://bootlin.com/
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [OE-core] [PATCH v9 0/3] pkg-database and systemd-sysext image
2024-05-21 10:08 ` Alexander Kanavin
@ 2024-05-22 10:59 ` SCHNEIDER Johannes
0 siblings, 0 replies; 8+ messages in thread
From: SCHNEIDER Johannes @ 2024-05-22 10:59 UTC (permalink / raw
To: Alexander Kanavin
Cc: Alexandre Belloni, openembedded-core@lists.openembedded.org,
richard.purdie@linuxfoundation.org
Hoi,
> If you scroll further up, there's:
ah... stupid me relied on the search function in the webui (which caches only part of the output...) instead of looking at/searching through the raw log; sorry!
> Also note that this happened only on the arm host, and not on x86
hosts. Which does make it tricky to reproduce, but at least gives a
lead:
curiously if i use an "actual arm host" = a debian arm64 schroot (see below),
building and running the tests in there also succeeds
-> so my current lead is that there is sth "wrong" with the qemu disk-image :-S
i'll have to pick apart how the yocto-autobuilder builds itself qemu-arm, a disk image and then runs the tests inside of that... convoluted :-D
since this most likely ends up in a special case/exception inside the test; would excluding that machine+test be an option?
gruß
Johannes
debian-arm schroot:
#> apt install qemu-user-static
#> mkdir -p debian-arm64/usr/bin/; cp /usr/bin/qemu-arm-static debian-arm64/usr/bin/
#> debootstrap --arch=arm64 bookworm debian-arm64 http://deb.debian.org/debian/
#chroot> apt install -y gawk wget git diffstat unzip texinfo gcc build-essential chrpath socat cpio python3 python3-pip python3-pexpect xz-utils debianutils iputils-ping python3-git python3-jinja2 python3-subunit zstd liblz4-tool file locales libacl1
#chroot /tmp/openembedded-core/build> SDKMACHINE=aarch64 MACHINE=qemuarm64 PACKAGE_CLASSES=package_rpm oe-selftest --verbose -r imagefeatures.ImageFeatures.test_image_gen_pkgdbfs_ipk
<SNIP>
2024-05-22 05:15:52,357 - oe-selftest - DEBUG - running cmd: /tmp/openembedded-core/build-st/tmp-glibc/sysroots/aarch64/usr/bin/opkg --volatile-cache --offline-root=/tmp/openembedded-core/build-st/tmp-glibc/deploy/images/qemuarm64 --add-arch qemuarm64:11 --add-arch qemuarm64:12 list-installed
<SNIP>
2024-05-22 05:16:06,765 - oe-selftest - INFO - oe-selftest - OK - All required tests passed (successes=1, skipped=0, failures=0, errors=0)
________________________________________
From: Alexander Kanavin <alex.kanavin@gmail.com>
Sent: Tuesday, May 21, 2024 12:08
To: SCHNEIDER Johannes
Cc: Alexandre Belloni; openembedded-core@lists.openembedded.org; richard.purdie@linuxfoundation.org
Subject: Re: [OE-core] [PATCH v9 0/3] pkg-database and systemd-sysext image
This email is not from Hexagon’s Office 365 instance. Please be careful while clicking links, opening attachments, or replying to this email.
If you scroll further up, there's:
2024-05-16 22:05:57,101 - oe-selftest - INFO - FAIL:
imagefeatures.ImageFeatures.test_image_gen_pkgdbfs_ipk
(subunit.RemotedTestCase)
2024-05-16 22:05:57,102 - oe-selftest - INFO -
----------------------------------------------------------------------
2024-05-16 22:05:57,102 - oe-selftest - INFO -
testtools.testresult.real._StringException: Traceback (most recent
call last):
File "/home/pokybuild/yocto-worker/oe-selftest-armhost/build/meta/lib/oeqa/selftest/cases/imagefeatures.py",
line 344, in test_image_gen_pkgdbfs_ipk
result = runCmd(cmd)
File "/home/pokybuild/yocto-worker/oe-selftest-armhost/build/meta/lib/oeqa/utils/commands.py",
line 212, in runCmd
raise AssertionError("Command '%s' returned non-zero exit status
%d:\n%s" % (command, result.status, exc_output))
AssertionError: Command
'/home/pokybuild/yocto-worker/oe-selftest-armhost/build/build-st-4137794/tmp/sysroots/aarch64/usr/bin/opkg
--volatile-cache
--offline-root=/home/pokybuild/yocto-worker/oe-selftest-armhost/build/build-st-4137794/tmp/deploy/images/qemuarm64
--add-arch qemuarm64:11 --add-arch qemuarm64:12 list-installed'
returned non-zero exit status 255:
Also note that this happened only on the arm host, and not on x86
hosts. Which does make it tricky to reproduce, but at least gives a
lead:
https://autobuilder.yoctoproject.org/typhoon/#/builders/83/builds/6921
Alex
On Sat, 18 May 2024 at 08:30, SCHNEIDER Johannes
<johannes.schneider@leica-geosystems.com> wrote:
>
> Hoi Alexandre,
>
>
> is there a way to get more output out of the autobuilder?
> the logs only tell "that" the test failed, not "why" = which assertion was missed
>
> if i run the test locally with the differing variables set like this:
> #> SDKMACHINE=aarch64 MACHINE=qemuarm64 oe-selftest --verbose -r imagefeatures.ImageFeatures.test_image_gen_pkgdbfs_ipk
>
> it succeeds :-S
> ... ideas/thoughts?
>
>
> gruß
> Johannes
>
> ________________________________________
> From: Alexandre Belloni <alexandre.belloni@bootlin.com>
> Sent: Friday, May 17, 2024 20:33
> To: SCHNEIDER Johannes
> Cc: openembedded-core@lists.openembedded.org; richard.purdie@linuxfoundation.org; alex.kanavin@gmail.com
> Subject: Re: [OE-core] [PATCH v9 0/3] pkg-database and systemd-sysext image
>
> This email is not from Hexagon’s Office 365 instance. Please be careful while clicking links, opening attachments, or replying to this email.
>
>
> Hello,
>
> deb and rpm seem successful but ipk still fails:
>
> https://autobuilder.yoctoproject.org/typhoon/#/builders/127/builds/3335/steps/14/logs/stdio
>
> On 16/05/2024 00:34:57+0200, Johannes Schneider via lists.openembedded.org wrote:
> > systemd-sysext allows to overlay another image (or multiple) ontop of
> > a "base-image" = the current rootfs, via the use of overlayfs; to add
> > tools and features meant for development purposes.
> >
> > To quote the documentation on systemd-sysext:
> > " ...addition in order to make debugging/development easier). System
> > extension images should not be misunderstood as a generic software
> > packaging framework, ..."
> >
> > To build a lean image, that only holds packages that are not already
> > part of the base-image, a snapshot of the package-database is taken
> > after the installation of the base-rootfs is done, and picked up again
> > when collecting the rootfs of such a extension image.
> >
> > with all this in place an example usage could look like this:
> > some-core-image.bb
> > inherit core-image
> > IMAGE_GEN_PKGDBFS = "1"
> >
> > extending-image.bb
> > inherit image-sysext
> > IMAGE_FSTYPES = "squashfs"
> > IMAGE_BASE_PKGDB = "some-core-image"
> > # the above pointing at a package-db similar to:
> > # build/deploy/images/$MACHINE/some-core-image-$MACHINE-20240210172305-pkgdb.rootfs.tar.gz
> >
> > then on the device, running some-core-image, with the extension image placed at FN:
> > $> ln -s "$FN" /run/extensions/$(basename $FN).raw
> > $> systemd-sysext list
> > $> SYSTEMD_LOG_LEVEL=debug systemd-sysext merge
> >
> > As long as the VERSION_ID of the extension image matches the os-release
> > in the base image, the above commands return sucessfully;
> > for details on the compativility check see the docs for systemd-sysext.
> >
> > =========
> >
> > changes with v2:
> > rebase from 'kirkstone' onto 'master'
> >
> > changes with v3:
> > incorporate review suggestions for simplification
> > add task dependency handling
> > add oe-selftest for the pkgdb handling
> > add variable documentation and
> > some more comments, and examples in the commit-msg
> >
> > changes with v4:
> > rebase onto 'master' => no functional changes
> > fixed patchtest findings
> >
> > changes with v5:
> > rebase onto 'master'
> > add '.sysext' to the deployed symlink name
> > sidenote on the tests and autobuilder failure: run locally they succeed, e.g.:
> > #> oe-selftest --verbose -r imagefeatures.ImageFeatures.test_image_gen_pkgdbfs
> >
> > changes with v6:
> > tests: restructure to call 'bitbake' only once in the testcase itself
> > (in hopes of solving the autobuilder problem; local test runs succeed)
> >
> > changes with v7:
> > tests: undo the restructuring of done in v6, in favour of explicitly forcing steps:
> > '-c rootfs -f'. since concurrency of the autobuilder, and reusing artifacts/sstate-caching is probably the issue?
> >
> > changes with v8:
> > tests: fixed missing feature=PACKAGE_CLASSES during 'test_image_gen_pkgdbfs'
> > which is now split into three, one for each of the three available packagemanager: ipk, deb, rpm
> >
> > changes with v9:
> > tests: fix singular failure in 'oe-selftest-armhost'
> > which was due to the built image not containing a 'kernel-image...', which the gen-pkgdb sanity
> > check expected, but 'packagegroup-core-boot' instead
>
> >
> > -=-=-=-=-=-=-=-=-=-=-=-
> > Links: You receive all messages sent to this group.
> > View/Reply Online (#199430): https://lists.openembedded.org/g/openembedded-core/message/199430
> > Mute This Topic: https://lists.openembedded.org/mt/106124365/3617179
> > Group Owner: openembedded-core+owner@lists.openembedded.org
> > Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [alexandre.belloni@bootlin.com]
> > -=-=-=-=-=-=-=-=-=-=-=-
> >
>
>
> --
> Alexandre Belloni, co-owner and COO, Bootlin
> Embedded Linux and Kernel engineering
> https://bootlin.com/
^ permalink raw reply [flat|nested] 8+ messages in thread
end of thread, other threads:[~2024-05-22 10:59 UTC | newest]
Thread overview: 8+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2024-05-15 22:34 [PATCH v9 0/3] pkg-database and systemd-sysext image Johannes Schneider
2024-05-15 22:34 ` [PATCH v9 1/3] image.bbclass/rootfs: archive and deploy package database Johannes Schneider
2024-05-15 22:34 ` [PATCH v9 2/3] image.bbclass/rootfs: set and unpack package-database Johannes Schneider
2024-05-15 22:35 ` [PATCH v9 3/3] classes: add a systemd-sysext image class Johannes Schneider
2024-05-17 18:33 ` [OE-core] [PATCH v9 0/3] pkg-database and systemd-sysext image Alexandre Belloni
2024-05-18 6:30 ` SCHNEIDER Johannes
2024-05-21 10:08 ` Alexander Kanavin
2024-05-22 10:59 ` SCHNEIDER Johannes
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.