* [PATCH 0/3] testimage/oeqa: Add ability to export tests to decouple them from the build system
@ 2014-02-03 19:22 Stefan Stanacar
2014-02-03 19:22 ` [PATCH 1/3] testimage: add ability to export tests Stefan Stanacar
` (2 more replies)
0 siblings, 3 replies; 4+ messages in thread
From: Stefan Stanacar @ 2014-02-03 19:22 UTC (permalink / raw
To: openembedded-core
Hello,
These patches add the ability to export and run tests outside of the build.
As tests used to query values from the data storage this exports parts of d
and makes them avaiable in a json file, so tests don't need to change to
make them exportable. Tests exported are the ones defined in TEST_SUITES regardless
of layer.
Starting the target (in the default case qemu) is still handled by the build system, as before,
so keep that in mind.
Simplest way to test this is with a qemu image that you manually start.
For an already build image use this in local.conf:
TEST_EXPORT_ONLY = "1"
TEST_TARGET = "simpleremote"
TEST_TARGET_IP = "192.168.7.2"
TEST_SERVER_IP = "192.168.7.1"
Export the tests: bitbake core-image-sato -c testimage
Then: runqemu core-image-sato
And: cd tmp/testimage/core-image-sato && ./runexported.py testdata.json
In the default case (TEST_EXPORT_ONLY = "0") nothing really changes
so no tests should start failing because of this.
Cheers,
Stefan
The following changes since commit 76f52a81f600fe7dfee0987db361e973b9ab71f8:
insane: Fix python function whitespace changes (2014-02-03 16:16:30 +0000)
are available in the git repository at:
git://git.yoctoproject.org/poky-contrib stefans/export
http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=stefans/export
Stefan Stanacar (3):
testimage: add ability to export tests
lib/oeqa: add module for running tests outside of the build system
oeqa/utils: targetbuild: don't use bb.fetch anymore
meta/classes/testimage.bbclass | 107 ++++++++++++++++++++++++----
meta/lib/oeqa/oetest.py | 17 ++---
meta/lib/oeqa/runexported.py | 140 +++++++++++++++++++++++++++++++++++++
meta/lib/oeqa/utils/targetbuild.py | 27 +++----
4 files changed, 246 insertions(+), 45 deletions(-)
create mode 100755 meta/lib/oeqa/runexported.py
--
1.8.5.3
^ permalink raw reply [flat|nested] 4+ messages in thread
* [PATCH 1/3] testimage: add ability to export tests
2014-02-03 19:22 [PATCH 0/3] testimage/oeqa: Add ability to export tests to decouple them from the build system Stefan Stanacar
@ 2014-02-03 19:22 ` Stefan Stanacar
2014-02-03 19:22 ` [PATCH 2/3] lib/oeqa: add module for running tests outside of the build system Stefan Stanacar
2014-02-03 19:22 ` [PATCH 3/3] oeqa/utils: targetbuild: don't use bb.fetch anymore Stefan Stanacar
2 siblings, 0 replies; 4+ messages in thread
From: Stefan Stanacar @ 2014-02-03 19:22 UTC (permalink / raw
To: openembedded-core
Add the ability to export the tests so that they can run independently of
the build system, as is required if you want to be able to hand the test
execution off to a scheduler.
Booting/deployment of the target is still handled by the build system,
as before, only the execution of the tests happens outside of the build system.
Tests exported are the ones defined in TEST_SUITES.
No tests have been changed as interesting parts of the data store have been
exported and tests can continue to query them as before. Small adjustments were made
for a couple of oeqa modules though.
[YOCTO #5613]
Signed-off-by: Stefan Stanacar <stefanx.stanacar@intel.com>
---
meta/classes/testimage.bbclass | 107 +++++++++++++++++++++++++++++++++++------
meta/lib/oeqa/oetest.py | 17 +++----
2 files changed, 98 insertions(+), 26 deletions(-)
diff --git a/meta/classes/testimage.bbclass b/meta/classes/testimage.bbclass
index 4777e14..75ab716 100644
--- a/meta/classes/testimage.bbclass
+++ b/meta/classes/testimage.bbclass
@@ -25,15 +25,16 @@
TEST_LOG_DIR ?= "${WORKDIR}/testimage"
+TEST_EXPORT_DIR ?= "${TMPDIR}/testimage/${PN}"
+TEST_EXPORT_ONLY ?= "0"
+
DEFAULT_TEST_SUITES = "ping auto"
DEFAULT_TEST_SUITES_pn-core-image-minimal = "ping"
DEFAULT_TEST_SUITES_pn-core-image-sato = "ping ssh df connman syslog xorg scp vnc date rpm smart dmesg python"
DEFAULT_TEST_SUITES_pn-core-image-sato-sdk = "ping ssh df connman syslog xorg scp vnc date perl ldd gcc rpm smart kernelmodule dmesg python"
-
TEST_SUITES ?= "${DEFAULT_TEST_SUITES}"
TEST_QEMUBOOT_TIMEOUT ?= "1000"
-
TEST_TARGET ?= "qemu"
TEST_TARGET_IP ?= ""
TEST_SERVER_IP ?= ""
@@ -85,6 +86,71 @@ def get_tests_list(d):
return testslist
+
+def exportTests(d,tc):
+ import json
+ import shutil
+ import pkgutil
+
+ exportpath = d.getVar("TEST_EXPORT_DIR", True)
+
+ savedata = {}
+ savedata["d"] = {}
+ savedata["target"] = {}
+ for key in tc.__dict__:
+ # special cases
+ if key != "d" and key != "target":
+ savedata[key] = getattr(tc, key)
+ savedata["target"]["ip"] = tc.target.ip or d.getVar("TEST_TARGET_IP", True)
+ savedata["target"]["server_ip"] = tc.target.server_ip or d.getVar("TEST_SERVER_IP", True)
+
+ keys = [ key for key in d.keys() if not key.startswith("_") and not key.startswith("BB") \
+ and not key.startswith("B_pn") and not key.startswith("do_") and not d.getVarFlag(key, "func")]
+ for key in keys:
+ try:
+ savedata["d"][key] = d.getVar(key, True)
+ except bb.data_smart.ExpansionError:
+ # we don't care about those anyway
+ pass
+
+ with open(os.path.join(exportpath, "testdata.json"), "w") as f:
+ json.dump(savedata, f, skipkeys=True, indent=4, sort_keys=True)
+
+ # now start copying files
+ # we'll basically copy everything under meta/lib/oeqa, with these exceptions
+ # - oeqa/targetcontrol.py - not needed
+ # - oeqa/selftest - something else
+ # That means:
+ # - all tests from oeqa/runtime defined in TEST_SUITES (including from other layers)
+ # - the contents of oeqa/utils and oeqa/runtime/files
+ # - oeqa/oetest.py and oeqa/runexport.py (this will get copied to exportpath not exportpath/oeqa)
+ # - __init__.py files
+ bb.utils.mkdirhier(os.path.join(exportpath, "oeqa/runtime/files"))
+ bb.utils.mkdirhier(os.path.join(exportpath, "oeqa/utils"))
+ # copy test modules, this should cover tests in other layers too
+ for t in tc.testslist:
+ mod = pkgutil.get_loader(t)
+ shutil.copy2(mod.filename, os.path.join(exportpath, "oeqa/runtime"))
+ # copy __init__.py files
+ oeqadir = pkgutil.get_loader("oeqa").filename
+ shutil.copy2(os.path.join(oeqadir, "__init__.py"), os.path.join(exportpath, "oeqa"))
+ shutil.copy2(os.path.join(oeqadir, "runtime/__init__.py"), os.path.join(exportpath, "oeqa/runtime"))
+ # copy oeqa/oetest.py and oeqa/runexported.py
+ shutil.copy2(os.path.join(oeqadir, "oetest.py"), os.path.join(exportpath, "oeqa"))
+ shutil.copy2(os.path.join(oeqadir, "runexported.py"), exportpath)
+ # copy oeqa/utils/*.py
+ for root, dirs, files in os.walk(os.path.join(oeqadir, "utils")):
+ for f in files:
+ if f.endswith(".py"):
+ shutil.copy2(os.path.join(root, f), os.path.join(exportpath, "oeqa/utils"))
+ # copy oeqa/runtime/files/*
+ for root, dirs, files in os.walk(os.path.join(oeqadir, "runtime/files")):
+ for f in files:
+ shutil.copy2(os.path.join(root, f), os.path.join(exportpath, "oeqa/runtime/files"))
+
+ bb.plain("Exported tests to: %s" % exportpath)
+
+
def testimage_main(d):
import unittest
import os
@@ -94,7 +160,11 @@ def testimage_main(d):
from oeqa.targetcontrol import get_target_controller
pn = d.getVar("PN", True)
+ export = oe.utils.conditional("TEST_EXPORT_ONLY", "1", True, False, d)
bb.utils.mkdirhier(d.getVar("TEST_LOG_DIR", True))
+ if export:
+ bb.utils.remove(d.getVar("TEST_EXPORT_DIR", True), recurse=True)
+ bb.utils.mkdirhier(d.getVar("TEST_EXPORT_DIR", True))
# tests in TEST_SUITES become required tests
# they won't be skipped even if they aren't suitable for a image (like xorg for minimal)
@@ -105,13 +175,18 @@ def testimage_main(d):
# the robot dance
target = get_target_controller(d)
- class TestContext:
+ class TestContext(object):
def __init__(self):
self.d = d
self.testslist = testslist
self.testsrequired = testsrequired
self.filesdir = os.path.join(os.path.dirname(os.path.abspath(oeqa.runtime.__file__)),"files")
self.target = target
+ self.imagefeatures = d.getVar("IMAGE_FEATURES", True).split()
+ self.distrofeatures = d.getVar("DISTRO_FEATURES", True).split()
+ manifest = os.path.join(d.getVar("DEPLOY_DIR_IMAGE", True), d.getVar("IMAGE_LINK_NAME", True) + ".manifest")
+ with open(manifest) as f:
+ self.pkgmanifest = f.read()
# test context
tc = TestContext()
@@ -129,19 +204,21 @@ def testimage_main(d):
try:
target.start()
- # run tests and get the results
- starttime = time.time()
- result = runTests(tc)
- stoptime = time.time()
- if result.wasSuccessful():
- bb.plain("%s - Ran %d test%s in %.3fs" % (pn, result.testsRun, result.testsRun != 1 and "s" or "", stoptime - starttime))
- msg = "%s - OK - All required tests passed" % pn
- skipped = len(result.skipped)
- if skipped:
- msg += " (skipped=%d)" % skipped
- bb.plain(msg)
+ if export:
+ exportTests(d,tc)
else:
- raise bb.build.FuncFailed("%s - FAILED - check the task log and the ssh log" % pn )
+ starttime = time.time()
+ result = runTests(tc)
+ stoptime = time.time()
+ if result.wasSuccessful():
+ bb.plain("%s - Ran %d test%s in %.3fs" % (pn, result.testsRun, result.testsRun != 1 and "s" or "", stoptime - starttime))
+ msg = "%s - OK - All required tests passed" % pn
+ skipped = len(result.skipped)
+ if skipped:
+ msg += " (skipped=%d)" % skipped
+ bb.plain(msg)
+ else:
+ raise bb.build.FuncFailed("%s - FAILED - check the task log and the ssh log" % pn )
finally:
target.stop()
diff --git a/meta/lib/oeqa/oetest.py b/meta/lib/oeqa/oetest.py
index 23a3e5d..0db6cb8 100644
--- a/meta/lib/oeqa/oetest.py
+++ b/meta/lib/oeqa/oetest.py
@@ -10,8 +10,6 @@
import os, re, mmap
import unittest
import inspect
-import bb
-from oeqa.utils.sshcontrol import SSHControl
def loadTests(tc):
@@ -31,15 +29,14 @@ def loadTests(tc):
def runTests(tc):
suite = loadTests(tc)
- bb.note("Test modules %s" % tc.testslist)
- bb.note("Found %s tests" % suite.countTestCases())
+ print("Test modules %s" % tc.testslist)
+ print("Found %s tests" % suite.countTestCases())
runner = unittest.TextTestRunner(verbosity=2)
result = runner.run(suite)
return result
-
class oeTest(unittest.TestCase):
longMessage = True
@@ -60,18 +57,16 @@ class oeTest(unittest.TestCase):
@classmethod
def hasPackage(self, pkg):
- manifest = os.path.join(oeTest.tc.d.getVar("DEPLOY_DIR_IMAGE", True), oeTest.tc.d.getVar("IMAGE_LINK_NAME", True) + ".manifest")
- with open(manifest) as f:
- data = f.read()
- if re.search(pkg, data):
+
+ if re.search(pkg, oeTest.tc.pkgmanifest):
return True
return False
@classmethod
def hasFeature(self,feature):
- if feature in oeTest.tc.d.getVar("IMAGE_FEATURES", True).split() or \
- feature in oeTest.tc.d.getVar("DISTRO_FEATURES", True).split():
+ if feature in oeTest.tc.imagefeatures or \
+ feature in oeTest.tc.distrofeatures:
return True
else:
return False
--
1.8.5.3
^ permalink raw reply related [flat|nested] 4+ messages in thread
* [PATCH 2/3] lib/oeqa: add module for running tests outside of the build system
2014-02-03 19:22 [PATCH 0/3] testimage/oeqa: Add ability to export tests to decouple them from the build system Stefan Stanacar
2014-02-03 19:22 ` [PATCH 1/3] testimage: add ability to export tests Stefan Stanacar
@ 2014-02-03 19:22 ` Stefan Stanacar
2014-02-03 19:22 ` [PATCH 3/3] oeqa/utils: targetbuild: don't use bb.fetch anymore Stefan Stanacar
2 siblings, 0 replies; 4+ messages in thread
From: Stefan Stanacar @ 2014-02-03 19:22 UTC (permalink / raw
To: openembedded-core
This script will run the exported tests outside of the build system.
Simplest way to test this is with a qemu image that you manually start.
For an already build image use this in local.conf:
TEST_EXPORT_ONLY = "1"
TEST_TARGET = "simpleremote"
TEST_TARGET_IP = "192.168.7.2"
TEST_SERVER_IP = "192.168.7.1"
Export the tests: bitbake core-image-sato -c testimage
Then: runqemu core-image-sato
And: cd build/tmp/testimage/core-image-sato
./runexported.py testdata.json
The contents of build/tmp/testimage/core-image-sato can be moved on another machine
as long as some paths are updated in the json.
The exported data contains paths to the build dir. We only care about DEPLOY_DIR/rpm (
if the rpm and smart tests are enabled), so running the tests on other machine
means that the user has to move the contents and call runexported with --deploy-dir PATH:
./runexported.py --deploy-dir /path/on/another/machine testdata.json
runexported.py accepts other arguments as well, see --help.
[YOCTO #5613]
Signed-off-by: Stefan Stanacar <stefanx.stanacar@intel.com>
---
meta/lib/oeqa/runexported.py | 140 +++++++++++++++++++++++++++++++++++++++++++
1 file changed, 140 insertions(+)
create mode 100755 meta/lib/oeqa/runexported.py
diff --git a/meta/lib/oeqa/runexported.py b/meta/lib/oeqa/runexported.py
new file mode 100755
index 0000000..e1b6642
--- /dev/null
+++ b/meta/lib/oeqa/runexported.py
@@ -0,0 +1,140 @@
+#!/usr/bin/env python
+
+
+# Copyright (C) 2013 Intel Corporation
+#
+# Released under the MIT license (see COPYING.MIT)
+
+# This script should be used outside of the build system to run image tests.
+# It needs a json file as input as exported by the build.
+# E.g for an already built image:
+#- export the tests:
+# TEST_EXPORT_ONLY = "1"
+# TEST_TARGET = "simpleremote"
+# TEST_TARGET_IP = "192.168.7.2"
+# TEST_SERVER_IP = "192.168.7.1"
+# bitbake core-image-sato -c testimage
+# Setup your target, e.g for qemu: runqemu core-image-sato
+# cd build/tmp/testimage/core-image-sato
+# ./runexported.py testdata.json
+
+import sys
+import os
+import time
+from optparse import OptionParser
+
+try:
+ import simplejson as json
+except ImportError:
+ import json
+
+sys.path.append(os.path.abspath(os.path.join(os.path.dirname(__file__), "oeqa")))
+
+from oeqa.oetest import runTests
+from oeqa.utils.sshcontrol import SSHControl
+
+# this isn't pretty but we need a fake target object
+# for running the tests externally as we don't care
+# about deploy/start we only care about the connection methods (run, copy)
+class FakeTarget(object):
+ def __init__(self, d):
+ self.connection = None
+ self.ip = None
+ self.server_ip = None
+ self.datetime = time.strftime('%Y%m%d%H%M%S',time.gmtime())
+ self.testdir = d.getVar("TEST_LOG_DIR", True)
+ self.pn = d.getVar("PN", True)
+
+ def exportStart(self):
+ self.sshlog = os.path.join(self.testdir, "ssh_target_log.%s" % self.datetime)
+ sshloglink = os.path.join(self.testdir, "ssh_target_log")
+ if os.path.islink(sshloglink):
+ os.unlink(sshloglink)
+ os.symlink(self.sshlog, sshloglink)
+ print("SSH log file: %s" % self.sshlog)
+ self.connection = SSHControl(self.ip, logfile=self.sshlog)
+
+ def run(self, cmd, timeout=None):
+ return self.connection.run(cmd, timeout)
+
+ def copy_to(self, localpath, remotepath):
+ return self.connection.copy_to(localpath, remotepath)
+
+ def copy_from(self, remotepath, localpath):
+ return self.connection.copy_from(remotepath, localpath)
+
+
+class MyDataDict(dict):
+ def getVar(self, key, unused = None):
+ return self.get(key, "")
+
+class TestContext(object):
+ def __init__(self):
+ self.d = None
+ self.target = None
+
+def main():
+
+ usage = "usage: %prog [options] <json file>"
+ parser = OptionParser(usage=usage)
+ parser.add_option("-t", "--target-ip", dest="ip", help="The IP address of the target machine. Use this to \
+ overwrite the value determined from TEST_TARGET_IP at build time")
+ parser.add_option("-s", "--server-ip", dest="server_ip", help="The IP address of this machine. Use this to \
+ overwrite the value determined from TEST_SERVER_IP at build time.")
+ parser.add_option("-d", "--deploy-dir", dest="deploy_dir", help="Full path to the package feeds, that this \
+ the contents of what used to be DEPLOY_DIR on the build machine. If not specified it will use the value \
+ specified in the json if that directory actually exists or it will error out.")
+ parser.add_option("-l", "--log-dir", dest="log_dir", help="This sets the path for TEST_LOG_DIR. If not specified \
+ the current dir is used. This is used for usually creating a ssh log file and a scp test file.")
+
+ (options, args) = parser.parse_args()
+ if len(args) != 1:
+ parser.error("Incorrect number of arguments. The one and only argument should be a json file exported by the build system")
+
+ with open(args[0], "r") as f:
+ loaded = json.load(f)
+
+ if options.ip:
+ loaded["target"]["ip"] = options.ip
+ if options.server_ip:
+ loaded["target"]["server_ip"] = options.server_ip
+
+ d = MyDataDict()
+ for key in loaded["d"].keys():
+ d[key] = loaded["d"][key]
+
+ if options.log_dir:
+ d["TEST_LOG_DIR"] = options.log_dir
+ else:
+ d["TEST_LOG_DIR"] = os.path.abspath(os.path.dirname(__file__))
+ if options.deploy_dir:
+ d["DEPLOY_DIR"] = options.deploy_dir
+ else:
+ if not os.path.isdir(d["DEPLOY_DIR"]):
+ raise Exception("The path to DEPLOY_DIR does not exists: %s" % d["DEPLOY_DIR"])
+
+
+ target = FakeTarget(d)
+ for key in loaded["target"].keys():
+ setattr(target, key, loaded["target"][key])
+
+ tc = TestContext()
+ setattr(tc, "d", d)
+ setattr(tc, "target", target)
+ for key in loaded.keys():
+ if key != "d" and key != "target":
+ setattr(tc, key, loaded[key])
+
+ target.exportStart()
+ runTests(tc)
+
+ return 0
+
+if __name__ == "__main__":
+ try:
+ ret = main()
+ except Exception:
+ ret = 1
+ import traceback
+ traceback.print_exc(5)
+ sys.exit(ret)
--
1.8.5.3
^ permalink raw reply related [flat|nested] 4+ messages in thread
* [PATCH 3/3] oeqa/utils: targetbuild: don't use bb.fetch anymore
2014-02-03 19:22 [PATCH 0/3] testimage/oeqa: Add ability to export tests to decouple them from the build system Stefan Stanacar
2014-02-03 19:22 ` [PATCH 1/3] testimage: add ability to export tests Stefan Stanacar
2014-02-03 19:22 ` [PATCH 2/3] lib/oeqa: add module for running tests outside of the build system Stefan Stanacar
@ 2014-02-03 19:22 ` Stefan Stanacar
2 siblings, 0 replies; 4+ messages in thread
From: Stefan Stanacar @ 2014-02-03 19:22 UTC (permalink / raw
To: openembedded-core
When running tests outside of the build system we can't use
bb.fetch anymore. It was nice but tests and their modules
need to rely on the data storage only as that gets exported.
This module is used by the oeqa/runtime/build* tests.
Signed-off-by: Stefan Stanacar <stefanx.stanacar@intel.com>
---
meta/lib/oeqa/utils/targetbuild.py | 27 ++++++++-------------------
1 file changed, 8 insertions(+), 19 deletions(-)
diff --git a/meta/lib/oeqa/utils/targetbuild.py b/meta/lib/oeqa/utils/targetbuild.py
index 9b2cf53..77181b1 100644
--- a/meta/lib/oeqa/utils/targetbuild.py
+++ b/meta/lib/oeqa/utils/targetbuild.py
@@ -4,11 +4,9 @@
# Provides a class for automating build tests for projects
-from oeqa.oetest import oeRuntimeTest
-import bb.fetch2
-import bb.data
import os
import re
+import subprocess
class TargetBuildProject():
@@ -16,26 +14,16 @@ class TargetBuildProject():
def __init__(self, target, uri, foldername=None):
self.target = target
self.uri = uri
- self.targetdir = "/home/root/"
-
- self.localdata = bb.data.createCopy(oeRuntimeTest.tc.d)
- bb.data.update_data(self.localdata)
-
- if not foldername:
- self.archive = os.path.basename(uri)
- self.fname = re.sub(r'.tar.bz2|tar.gz$', '', self.archive)
- else:
+ self.targetdir = "~/"
+ self.archive = os.path.basename(uri)
+ self.localarchive = "/tmp/" + self.archive
+ self.fname = re.sub(r'.tar.bz2|tar.gz$', '', self.archive)
+ if foldername:
self.fname = foldername
def download_archive(self):
- try:
- self.localdata.delVar("BB_STRICT_CHECKSUM")
- fetcher = bb.fetch2.Fetch([self.uri], self.localdata)
- fetcher.download()
- self.localarchive = fetcher.localpath(self.uri)
- except bb.fetch2.BBFetchException:
- raise Exception("Failed to download archive: %s" % self.uri)
+ subprocess.check_call("wget -O %s %s" % (self.localarchive, self.uri), shell=True)
(status, output) = self.target.copy_to(self.localarchive, self.targetdir)
if status != 0:
@@ -61,3 +49,4 @@ class TargetBuildProject():
def clean(self):
self.target.run('rm -rf %s' % self.targetdir)
+ subprocess.call('rm -f %s' % self.localarchive, shell=True)
--
1.8.5.3
^ permalink raw reply related [flat|nested] 4+ messages in thread
end of thread, other threads:[~2014-02-03 19:23 UTC | newest]
Thread overview: 4+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2014-02-03 19:22 [PATCH 0/3] testimage/oeqa: Add ability to export tests to decouple them from the build system Stefan Stanacar
2014-02-03 19:22 ` [PATCH 1/3] testimage: add ability to export tests Stefan Stanacar
2014-02-03 19:22 ` [PATCH 2/3] lib/oeqa: add module for running tests outside of the build system Stefan Stanacar
2014-02-03 19:22 ` [PATCH 3/3] oeqa/utils: targetbuild: don't use bb.fetch anymore Stefan Stanacar
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.