From: Joshua Watt <jpewhacker@gmail.com>
To: bitbake-devel@lists.openembedded.org
Cc: Tobias Hagelborn <tobias.hagelborn@axis.com>,
Tobias Hagelborn <tobiasha@axis.com>,
Richard Purdie <richard.purdie@linuxfoundation.org>,
Joshua Watt <JPEWhacker@gmail.com>
Subject: [bitbake-devel][PATCH v2 8/8] bitbake: hashserv: Postgres adaptations for ignoring duplicate inserts
Date: Sun, 18 Feb 2024 15:59:53 -0700 [thread overview]
Message-ID: <20240218225953.2997239-9-JPEWhacker@gmail.com> (raw)
In-Reply-To: <20240218225953.2997239-1-JPEWhacker@gmail.com>
From: Tobias Hagelborn <tobias.hagelborn@axis.com>
Hash Equivalence server performs unconditional insert also of duplicate
hash entries. This causes excessive error log entries in Postgres.
Rather ignore the duplicate inserts.
The alternate behavior should be isolated to the postgres
engine type.
Signed-off-by: Tobias Hagelborn <tobiasha@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
---
bitbake/lib/hashserv/sqlalchemy.py | 53 +++++++++++++++++++++---------
1 file changed, 38 insertions(+), 15 deletions(-)
diff --git a/bitbake/lib/hashserv/sqlalchemy.py b/bitbake/lib/hashserv/sqlalchemy.py
index 0e28d738f5a..fc3ae3d3396 100644
--- a/bitbake/lib/hashserv/sqlalchemy.py
+++ b/bitbake/lib/hashserv/sqlalchemy.py
@@ -33,6 +33,7 @@ from sqlalchemy import (
import sqlalchemy.engine
from sqlalchemy.orm import declarative_base
from sqlalchemy.exc import IntegrityError
+from sqlalchemy.dialects.postgresql import insert as postgres_insert
Base = declarative_base()
@@ -283,9 +284,7 @@ class Database(object):
async def unihash_exists(self, unihash):
async with self.db.begin():
result = await self._execute(
- select(UnihashesV3)
- .where(UnihashesV3.unihash == unihash)
- .limit(1)
+ select(UnihashesV3).where(UnihashesV3.unihash == unihash).limit(1)
)
return result.first() is not None
@@ -435,18 +434,30 @@ class Database(object):
return result.rowcount
async def insert_unihash(self, method, taskhash, unihash):
- try:
- async with self.db.begin():
- await self._execute(
- insert(UnihashesV3).values(
- method=method,
- taskhash=taskhash,
- unihash=unihash,
- gc_mark=self._get_config_subquery("gc-mark", ""),
- )
+ # Postgres specific ignore on insert duplicate
+ if self.engine.name == "postgresql":
+ statement = (
+ postgres_insert(UnihashesV3)
+ .values(
+ method=method,
+ taskhash=taskhash,
+ unihash=unihash,
+ gc_mark=self._get_config_subquery("gc-mark", ""),
)
+ .on_conflict_do_nothing(index_elements=("method", "taskhash"))
+ )
+ else:
+ statement = insert(UnihashesV3).values(
+ method=method,
+ taskhash=taskhash,
+ unihash=unihash,
+ gc_mark=self._get_config_subquery("gc-mark", ""),
+ )
- return True
+ try:
+ async with self.db.begin():
+ result = await self._execute(statement)
+ return result.rowcount != 0
except IntegrityError:
self.logger.debug(
"%s, %s, %s already in unihash database", method, taskhash, unihash
@@ -461,10 +472,22 @@ class Database(object):
if "created" in data and not isinstance(data["created"], datetime):
data["created"] = datetime.fromisoformat(data["created"])
+ # Postgres specific ignore on insert duplicate
+ if self.engine.name == "postgresql":
+ statement = (
+ postgres_insert(OuthashesV2)
+ .values(**data)
+ .on_conflict_do_nothing(
+ index_elements=("method", "taskhash", "outhash")
+ )
+ )
+ else:
+ statement = insert(OuthashesV2).values(**data)
+
try:
async with self.db.begin():
- await self._execute(insert(OuthashesV2).values(**data))
- return True
+ result = await self._execute(statement)
+ return result.rowcount != 0
except IntegrityError:
self.logger.debug(
"%s, %s already in outhash database", data["method"], data["outhash"]
--
2.34.1
prev parent reply other threads:[~2024-02-18 23:00 UTC|newest]
Thread overview: 15+ messages / expand[flat|nested] mbox.gz Atom feed top
2024-02-18 20:07 [bitbake-devel][PATCH 0/5] Implement parallel Query API Joshua Watt
2024-02-18 20:07 ` [bitbake-devel][PATCH 1/5] hashserv: sqlalchemy: Use _execute() helper Joshua Watt
2024-02-18 20:07 ` [bitbake-devel][PATCH 2/5] hashserv: Add unihash-exists API Joshua Watt
2024-02-18 20:07 ` [bitbake-devel][PATCH 3/5] asyncrpc: Add Client Pool object Joshua Watt
2024-02-18 20:07 ` [bitbake-devel][PATCH 4/5] hashserv: Add Client Pool Joshua Watt
2024-02-18 20:07 ` [bitbake-devel][PATCH 5/5] siggen: Add parallel query API Joshua Watt
2024-02-18 22:59 ` [bitbake-devel][PATCH v2 0/8] Implement parallel Query API Joshua Watt
2024-02-18 22:59 ` [bitbake-devel][PATCH v2 1/8] hashserv: Add Unihash Garbage Collection Joshua Watt
2024-02-18 22:59 ` [bitbake-devel][PATCH v2 2/8] hashserv: sqlalchemy: Use _execute() helper Joshua Watt
2024-02-18 22:59 ` [bitbake-devel][PATCH v2 3/8] hashserv: Add unihash-exists API Joshua Watt
2024-02-18 22:59 ` [bitbake-devel][PATCH v2 4/8] asyncrpc: Add Client Pool object Joshua Watt
2024-02-18 22:59 ` [bitbake-devel][PATCH v2 5/8] hashserv: Add Client Pool Joshua Watt
2024-02-18 22:59 ` [bitbake-devel][PATCH v2 6/8] siggen: Add parallel query API Joshua Watt
2024-02-18 22:59 ` [bitbake-devel][PATCH v2 7/8] siggen: Add parallel unihash exist API Joshua Watt
2024-02-18 22:59 ` Joshua Watt [this message]
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20240218225953.2997239-9-JPEWhacker@gmail.com \
--to=jpewhacker@gmail.com \
--cc=bitbake-devel@lists.openembedded.org \
--cc=richard.purdie@linuxfoundation.org \
--cc=tobias.hagelborn@axis.com \
--cc=tobiasha@axis.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).