* [Buildroot] [PATCH 1/1] support/scripts/pkg-stats: clear multiprocessing pools after use
@ 2020-03-01 21:25 Titouan Christophe
2020-03-02 22:37 ` Peter Korsgaard
0 siblings, 1 reply; 2+ messages in thread
From: Titouan Christophe @ 2020-03-01 21:25 UTC (permalink / raw
To: buildroot
During the CVE checking phase, we can still see a huge amount of
Python processes (actually 128) running on the host, even though
the CVE step is entirely ran in the main thread.
These are actually the worker processes spawned to check for the
packages URL statuses and the latest versions from release-monitoring.
This is because of an issue in Python's multiprocessing implementation:
https://bugs.python.org/issue34172
The problem was already there before the CVE matching step was
introduced, but because pkg-stat was terminating right after the
release-monitoring step, it went unnoticed.
Also, do not hold a reference to the multiprocessing pool from
the Package class, as this is not needed.
Signed-off-by: Titouan Christophe <titouan.christophe@railnova.eu>
---
support/scripts/pkg-stats | 7 +++++--
1 file changed, 5 insertions(+), 2 deletions(-)
diff --git a/support/scripts/pkg-stats b/support/scripts/pkg-stats
index 7721d98459..4f653a6ad9 100755
--- a/support/scripts/pkg-stats
+++ b/support/scripts/pkg-stats
@@ -402,11 +402,13 @@ def check_url_status_worker(url, url_status):
def check_package_urls(packages):
- Package.pool = Pool(processes=64)
+ pool = Pool(processes=64)
for pkg in packages:
- pkg.url_worker = pkg.pool.apply_async(check_url_status_worker, (pkg.url, pkg.url_status))
+ pkg.url_worker = pool.apply_async(check_url_status_worker, (pkg.url, pkg.url_status))
for pkg in packages:
pkg.url_status = pkg.url_worker.get(timeout=3600)
+ del pkg.url_worker
+ pool.terminate()
def release_monitoring_get_latest_version_by_distro(pool, name):
@@ -479,6 +481,7 @@ def check_package_latest_version(packages):
results = worker_pool.map(check_package_latest_version_worker, (pkg.name for pkg in packages))
for pkg, r in zip(packages, results):
pkg.latest_version = r
+ worker_pool.terminate()
del http_pool
--
2.24.1
^ permalink raw reply related [flat|nested] 2+ messages in thread
* [Buildroot] [PATCH 1/1] support/scripts/pkg-stats: clear multiprocessing pools after use
2020-03-01 21:25 [Buildroot] [PATCH 1/1] support/scripts/pkg-stats: clear multiprocessing pools after use Titouan Christophe
@ 2020-03-02 22:37 ` Peter Korsgaard
0 siblings, 0 replies; 2+ messages in thread
From: Peter Korsgaard @ 2020-03-02 22:37 UTC (permalink / raw
To: buildroot
>>>>> "Titouan" == Titouan Christophe <titouan.christophe@railnova.eu> writes:
> During the CVE checking phase, we can still see a huge amount of
> Python processes (actually 128) running on the host, even though
> the CVE step is entirely ran in the main thread.
> These are actually the worker processes spawned to check for the
> packages URL statuses and the latest versions from release-monitoring.
> This is because of an issue in Python's multiprocessing implementation:
> https://bugs.python.org/issue34172
> The problem was already there before the CVE matching step was
> introduced, but because pkg-stat was terminating right after the
> release-monitoring step, it went unnoticed.
> Also, do not hold a reference to the multiprocessing pool from
> the Package class, as this is not needed.
> Signed-off-by: Titouan Christophe <titouan.christophe@railnova.eu>
Committed, thanks.
--
Bye, Peter Korsgaard
^ permalink raw reply [flat|nested] 2+ messages in thread
end of thread, other threads:[~2020-03-02 22:37 UTC | newest]
Thread overview: 2+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2020-03-01 21:25 [Buildroot] [PATCH 1/1] support/scripts/pkg-stats: clear multiprocessing pools after use Titouan Christophe
2020-03-02 22:37 ` Peter Korsgaard
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.