From: Jan-Simon Moeller <dl9pf@gmx.de>
To: michael.opdenacker@bootlin.com
Cc: bitbake-devel@lists.openembedded.org
Subject: Re: [bitbake-devel] [PATCH v6 0/8] prserv: add support for an "upstream" server
Date: Tue, 30 Apr 2024 22:31:47 +0200 [thread overview]
Message-ID: <5793552.DvuYhMxLoT@monster> (raw)
In-Reply-To: <20240430171512.936371-1-michael.opdenacker@bootlin.com>
Reviewed-by: Jan-Simon Möller <dl9pf@gmx.de>
Am Dienstag, 30. April 2024, 19:15:04 CEST schrieb Michael Opdenacker via
lists.openembedded.org:
> From: Michael Opdenacker <michael.opdenacker@bootlin.com>
>
> This makes it possible to customize an "upstream" distribution
> by modifying local packages. If the "upstream" package bears
> revision "x", the local one will have revision "x.y", this
> having priority over the upstream one.
>
> Multiple levels of upstream servers are supported, so "x.y.z" revisions
> are possible too.
>
> This version also supports multiple servers sharing the same database.
>
> This also includes BitBake selftests to check the expected
> functionality and detect future regressions.
>
> Note that database import and export functions aren't tested yet.
>
> ---
>
> Changes in V6:
>
> - Harden the store_value() code to protect
> it against storing the same value twice.
>
> - Simplify the importone() hook, removing
> its unnecessary "history" parameter.
>
> - This fixes all errors in "oe-selftest -r prservice",
> once this OE-core patch is merged:
>
> https://lore.kernel.org/openembedded-core/20240429194656.655509-1-michael.o
> pdenacker@bootlin.com/T/#u
>
> - Make tests work in temporary directories.
> Suggested by Joshua Watt
>
> - Remove first version of readonly server tests
> which depended on the output of a previous test.
> One implementation of readonly server tests remains.
>
> - Merge the upstream and read-only tests so that
> the read-only test part can reuse the databases
> and servers from the upstream one. Initially thought
> that the setUp (and tearDown) hooks were called only
> once for each test class, instead of for each test.
> Thanks to Joshua Watt for making me understand this.
>
> - Merged two more tests and simplified their naming
> as, with one exception, the order of tests doesn't
> matter any more.
>
> - Avoid a possible race condition at database
> creation time. Suggested by Joshua Watt.
>
> Changes in V5:
>
> - Change the way the database is accessed and modified.
> As done in the Hash Server code, the database is not
> accessed through the cursor API.
>
> This makes it possible to have multiple servers accessing
> the same database, typically one read-write server that
> is fed by the build machine, and one read-only server that
> is used to publish PR data to downstream users.
>
> This switched sqlite3 to "WAL" journaling mode, and to
> "OFF" synchronous mode.
>
> - Fix revision ordering issues, now computing the min and max
> values through dedicated Python functions, instead of letting
> the database do this with just string comparison, which
> was wrong in our case. Typically, "1.20" is greater for us
> than "1.3".
>
> - Add BitBake selftests, in particular to make sure revision
> ordering is correct and the maximum and minimum values are
> correctly computed.
>
> - An issue remains: the BitBake selftests that test database
> access through a read-only server while a read-write server
> is still running, still fail, because the database changes
> are apparently not stored to disk.
>
> FAIL: test_3b_readonly (prserv.tests.PRUpstreamTests)
>
> Changes in V4:
>
> - Add BitBake selftests for the legacy and new PR server features
> (database, client, server, support for upstream server, read-only mode,
> "history" and "no history" modes) To run only these tests:
> bitbake-selftest prserv.tests
>
> - Pass the "history" mode through the client requests, instead
> of storing it (globally) in the database name.
>
> The PR database is now called "PRMAIN" instead of "PRMAIN_nohist".
> This should cause a regression for builds which already
> have a PR database.
>
> - Fixes for "history" modes:
> - Allow to store multiple PR values for the same checksum,
> needed for the "no history" mode.
> - Make the "history" mode return the minimum stored
> PR value.
>
> - Fixes and code reorganization for issues uncovered by the tests.
>
> - Update the server version to "2.0.0"
>
> Changes in V3:
>
> - Revert the commit removing the so far unused "hist" mode, which
> we wish to keep for binary reproducibility sake.
>
> - Simplification of get_value() function to take
> both "hist" and "nohist" modes with the same shared code.
>
> - Add "history" parameter to the "getPR" request,
> so that the client can ask for the mode of its choice.
> This will also make it possible to implement tests
> for both modes.
>
> Note that more requests ("export", "import"...)
> will also need a "history" parameter, in a future version,
> after the first tests are implemented.
>
> - Several bug fixes.
>
> - Put all the new features at the tip of the branch,
> to make the cleanup commits easier to merged.
>
> Changes in V2:
>
> - Add this new commit:
> prserv: remove unused "hist" mode in the database backend
>
> - Squash commit "prserv: fix read_only test" into
> commit "prserv: simplify the PRServerClient() interface"
> (Reported by Richard Purdie)
>
> - Fix the code to support increasing "x.y.z" values, thus
> supporting several levels of upstream servers.
>
> - db.py: remove duplicate definition of find_max_value() function in db.py
>
> - prserv.py: remove tabs before comments (Python didn't complain)
>
> - db.py: now stores the revision ("value") as TEXT.
> This way we can store "1.0" without having it transformed to "1"
> when the default type was INTEGER.
>
> - This allows to fix a regression when the first packages were created
> with 'r0.1' instead of 'r0.0' initially.
>
> - find_max_value: now returns None instead of '0' when no value is found
> Before we couldn't tell the difference between a '0'
> max value and the absence of such a value.
>
> Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
> Cc: Joshua Watt <JPEWhacker@gmail.com>
> Cc: Tim Orling <ticotimo@gmail.com>
>
> Michael Opdenacker (8):
> prserv: declare "max_package_pr" client hook
> prserv: move code from __init__ to bitbake-prserv
> prserv: add "upstream" server support
> prserv: enable database sharing
> prserv: avoid possible race condition in database code
> prserv: store_value() improvements
> prserv: import simplification
> prserv: add bitbake selftests
>
> bin/bitbake-prserv | 26 ++-
> bin/bitbake-selftest | 2 +
> lib/prserv/__init__.py | 97 ++++++++-
> lib/prserv/client.py | 15 +-
> lib/prserv/db.py | 452 ++++++++++++++++++-----------------------
> lib/prserv/serv.py | 139 ++++++++++---
> lib/prserv/tests.py | 386 +++++++++++++++++++++++++++++++++++
> 7 files changed, 816 insertions(+), 301 deletions(-)
> create mode 100644 bitbake/lib/prserv/tests.py
prev parent reply other threads:[~2024-04-30 20:31 UTC|newest]
Thread overview: 11+ messages / expand[flat|nested] mbox.gz Atom feed top
2024-04-30 17:15 [PATCH v6 0/8] prserv: add support for an "upstream" server michael.opdenacker
2024-04-30 17:15 ` [PATCH v6 1/8] prserv: declare "max_package_pr" client hook michael.opdenacker
2024-04-30 17:15 ` [PATCH v6 2/8] prserv: move code from __init__ to bitbake-prserv michael.opdenacker
2024-04-30 17:15 ` [PATCH v6 3/8] prserv: add "upstream" server support michael.opdenacker
2024-04-30 17:15 ` [PATCH v6 4/8] prserv: enable database sharing michael.opdenacker
2024-04-30 20:29 ` [bitbake-devel] " Jan-Simon Moeller
2024-04-30 17:15 ` [PATCH v6 5/8] prserv: avoid possible race condition in database code michael.opdenacker
2024-04-30 17:15 ` [PATCH v6 6/8] prserv: store_value() improvements michael.opdenacker
2024-04-30 17:15 ` [PATCH v6 7/8] prserv: import simplification michael.opdenacker
2024-04-30 17:15 ` [PATCH v6 8/8] prserv: add bitbake selftests michael.opdenacker
2024-04-30 20:31 ` Jan-Simon Moeller [this message]
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=5793552.DvuYhMxLoT@monster \
--to=dl9pf@gmx.de \
--cc=bitbake-devel@lists.openembedded.org \
--cc=michael.opdenacker@bootlin.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).