From: ChenQi <Qi.Chen@windriver.com>
To: bitbake-devel@lists.openembedded.org
Subject: Re: [bitbake-devel][PATCH 1/3] fetch2/git.py: fix a corner case in try_premirror
Date: Sun, 18 Feb 2024 10:30:00 +0800 [thread overview]
Message-ID: <163e8e4c-60a1-912b-072a-89fadfe1e519@windriver.com> (raw)
In-Reply-To: <20240201033100.3617421-1-Qi.Chen@windriver.com>
ping
On 2/1/24 11:30, Chen Qi via lists.openembedded.org wrote:
> From: Chen Qi <Qi.Chen@windriver.com>
>
> For gitsm recipes, it's possible that some URL is used more than once.
> e.g.,
> A -> B:rev1 (B is a submodule of A)
> A -> C (C is a submodule of A)
> C -> B:rev2 (B is a submodule of C)
> A anc C are both using B as submodules, but on different revs.
> Now if we have:
> B:rev1 -> D
> B:rev2 -> E
> Then, the mirror will not be fully used.
> Say we have all repo mirrors for A, B, C, D, E, then in theory it's not
> necessary to reach out to any network for downloading. But it's not the
> case. After downloading B(rev1) and its submodule D from mirrors, the fetch
> process continues to download C, thus B(rev2) and E. Now it finds that B
> needs an update because its submodule E needs an update. Of course this is
> true because E is not downloaded yet. Now the problem comes to whether to
> use mirror or not. The git.py defines try_premirror to return 'False' when
> the ud.clonedir exists. As B has been cloned, the ud.clonedir exists and
> try_mirror returns False, resulting in not using mirror and going to upstream
> directly.
>
> We can see that the mirrors are not fully used. This is usually not problem,
> as the cost is only some network download. But in case the following two
> settings are there, we get errors.
> BB_NO_NETWORK = "0"
> BB_ALLOWED_NETWORKS = "*.some.allowed.domain"
> In such case, the gitsm recipe A will fail to fetch. Note that all contents
> that A needs are in mirrors and now it's failing to fetch. This is unexpected.
>
> Note that the different revs of the same repo in gitsm recipe is not the only
> way to reveal this problem. For example, there might be a recipe call B that
> uses B:rev3. Check the protobuf and grpc recipes as an example.
>
> For now, we can use the following steps to reproduce this issue. To be clear,
> the grpc recipe in meta-oe is now 1.60.0.
> 1. Add in local.conf:
> DL_DIR = "${TOPDIR}/downloads-premirror"
> bitbake grpc -c fetch
> 2. Comment out the DL_DIR setting in local.conf and add the following lines:
> PREMIRRORS:append = " \
> git://.*/.* git://${TOPDIR}/downloads-premirror/git2/MIRRORNAME;protocol=file \n \
> gitsm://.*/.* gitsm://${TOPDIR}/downloads-premirror/git2/MIRRORNAME;protocol=file \n \
> "
> 3. Set BB_NO_NETWORK = "1" and then 'bitbake grpc -c fetch'.
> This command succeeds and this shows that the premirror holds everything we need.
> 4. Add the following lines and then 'bitbake grpc -c fetch'.
> BB_NO_NETWORK = "0"
> BB_ALLOWED_NETWORKS = "*.some.domain"
>
> After step 4, the error message is as below:
> ERROR: grpc-1.60.0-r0 do_fetch: The URL: 'gitsm://github.com/protocolbuffers/protobuf.git;protocol=https;name=third_party/protobuf;subpath=third_party/protobuf;nobranch=1;lfs=True;bareclone=1;nobranch=1' is not trusted and cannot be used
>
> This patch fixes this problem by handling this corner case, that is, if the URL is
> not trusted from the settings of BB_NO_NETWORK and BB_ALLOWED_NETWORKS, then we should
> try premirrors because trying to reach upstream is destined to fail.
>
> Signed-off-by: Chen Qi <Qi.Chen@windriver.com>
> ---
> bitbake/lib/bb/fetch2/git.py | 6 ++++++
> 1 file changed, 6 insertions(+)
>
> diff --git a/bitbake/lib/bb/fetch2/git.py b/bitbake/lib/bb/fetch2/git.py
> index 0deeb5cee1..322a9366da 100644
> --- a/bitbake/lib/bb/fetch2/git.py
> +++ b/bitbake/lib/bb/fetch2/git.py
> @@ -87,6 +87,7 @@ from contextlib import contextmanager
> from bb.fetch2 import FetchMethod
> from bb.fetch2 import runfetchcmd
> from bb.fetch2 import logger
> +from bb.fetch2 import trusted_network
>
>
> sha1_re = re.compile(r'^[0-9a-f]{40}$')
> @@ -355,6 +356,11 @@ class Git(FetchMethod):
> # is not possible
> if bb.utils.to_boolean(d.getVar("BB_FETCH_PREMIRRORONLY")):
> return True
> + # If the url is not in trusted network, that is, BB_NO_NETWORK is set to 0
> + # and BB_ALLOWED_NETWORKS does not contain the host that ud.url uses, then
> + # we need to try premirrors first as using upstream is destined to fail.
> + if not trusted_network(d, ud.url):
> + return True
> if os.path.exists(ud.clonedir):
> return False
> return True
>
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> View/Reply Online (#15817): https://lists.openembedded.org/g/bitbake-devel/message/15817
> Mute This Topic: https://lists.openembedded.org/mt/104090271/7304865
> Group Owner: bitbake-devel+owner@lists.openembedded.org
> Unsubscribe: https://lists.openembedded.org/g/bitbake-devel/unsub [Qi.Chen@eng.windriver.com]
> -=-=-=-=-=-=-=-=-=-=-=-
>
prev parent reply other threads:[~2024-02-18 2:30 UTC|newest]
Thread overview: 6+ messages / expand[flat|nested] mbox.gz Atom feed top
2024-02-01 3:30 [bitbake-devel][PATCH 1/3] fetch2/git.py: fix a corner case in try_premirror Qi.Chen
2024-02-01 3:30 ` [bitbake-devel][PATCH 2/3] tests/fetch.py: add test case for using premirror in restricted network Qi.Chen
2024-02-01 3:31 ` [bitbake-devel][PATCH 3/3] fetch2/git.py: add comment in try_premirrors Qi.Chen
2024-02-01 11:15 ` Sam Liddicott
2024-02-02 4:24 ` ChenQi
2024-02-18 2:30 ` ChenQi [this message]
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=163e8e4c-60a1-912b-072a-89fadfe1e519@windriver.com \
--to=qi.chen@windriver.com \
--cc=bitbake-devel@lists.openembedded.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).