From: "Xing Xin via GitGitGadget" <gitgitgadget@gmail.com>
To: git@vger.kernel.org
Cc: Patrick Steinhardt <ps@pks.im>,
Karthik Nayak <karthik.188@gmail.com>,
blanet <bupt_xingxin@163.com>,
Xing Xin <xingxin.xx@bytedance.com>
Subject: [PATCH v8 1/3] bundle-uri: verify oid before writing refs
Date: Wed, 19 Jun 2024 04:07:31 +0000 [thread overview]
Message-ID: <d8fbde2dcd49d2a2a37e35f456830c51709cd743.1718770053.git.gitgitgadget@gmail.com> (raw)
In-Reply-To: <pull.1730.v8.git.1718770053.gitgitgadget@gmail.com>
From: Xing Xin <xingxin.xx@bytedance.com>
When using the bundle-uri mechanism with a bundle list containing
multiple interrelated bundles, we encountered a bug where tips from
downloaded bundles were not discovered, thus resulting in rather slow
clones. This was particularly problematic when employing the
"creationTokens" heuristic.
To reproduce this issue, consider a repository with a single branch
"main" pointing to commit "A". Firstly, create a base bundle with:
git bundle create base.bundle main
Then, add a new commit "B" on top of "A", and create an incremental
bundle for "main":
git bundle create incr.bundle A..main
Now, generate a bundle list with the following content:
[bundle]
version = 1
mode = all
heuristic = creationToken
[bundle "base"]
uri = base.bundle
creationToken = 1
[bundle "incr"]
uri = incr.bundle
creationToken = 2
A fresh clone with the bundle list above should result in a reference
"refs/bundles/main" pointing to "B" in the new repository. However, git
would still download everything from the server, as if it had fetched
nothing locally.
So why the "refs/bundles/main" is not discovered? After some digging I
found that:
1. Bundles in bundle list are downloaded to local files via
`bundle-uri.c:download_bundle_list` or via
`bundle-uri.c:fetch_bundles_by_token` for the "creationToken"
heuristic.
2. Each bundle is unbundled via `bundle-uri.c:unbundle_from_file`, which
is called by `bundle-uri.c:unbundle_all_bundles` or called within
`bundle-uri.c:fetch_bundles_by_token` for the "creationToken"
heuristic.
3. To get all prerequisites of the bundle, the bundle header is read
inside `bundle-uri.c:unbundle_from_file` to by calling
`bundle.c:read_bundle_header`.
4. Then it calls `bundle.c:unbundle`, which calls
`bundle.c:verify_bundle` to ensure the repository contains all the
prerequisites.
5. `bundle.c:verify_bundle` calls `parse_object`, which eventually
invokes `packfile.c:prepare_packed_git` or
`packfile.c:reprepare_packed_git`, filling
`raw_object_store->packed_git` and setting `packed_git_initialized`.
6. If `bundle.c:unbundle` succeeds, it writes refs via
`refs.c:refs_update_ref` with `REF_SKIP_OID_VERIFICATION` set. Here
bundle refs which can target arbitrary objects are written to the
repository.
7. Finally, in `fetch-pack.c:do_fetch_pack_v2`, the functions
`fetch-pack.c:mark_complete_and_common_ref` and
`fetch-pack.c:mark_tips` are called with `OBJECT_INFO_QUICK` set to
find local tips for negotiation. The `OBJECT_INFO_QUICK` flag
prevents `packfile.c:reprepare_packed_git` from being called,
resulting in failures to parse OIDs that reside only in the latest
bundle.
In the example above, when unbunding "incr.bundle", "base.pack" is added
to `packed_git` due to prerequisites verification. However, "B" cannot
be found for negotiation because it exists in "incr.pack", which is not
included in `packed_git`.
Fix the bug by removing `REF_SKIP_OID_VERIFICATION` flag when writing
bundle refs. When `refs.c:refs_update_ref` is called to write the
corresponding bundle refs, it triggers `refs.c:ref_transaction_commit`.
This, in turn, invokes `refs.c:ref_transaction_prepare`, which calls
`transaction_prepare` of the refs storage backend. For files backend, it
is `files-backend.c:files_transaction_prepare`, and for reftable
backend, it is `reftable-backend.c:reftable_be_transaction_prepare`.
Both functions eventually call `object.c:parse_object`, which can invoke
`packfile.c:reprepare_packed_git` to refresh `packed_git`. This ensures
that bundle refs point to valid objects and that all tips from bundle
refs are correctly parsed during subsequent negotiations.
A set of negotiation-related tests for cloning with bundle-uri has been
included to demonstrate that downloaded bundles are utilized to
accelerate fetching.
Additionally, another test has been added to show that bundles with
incorrect headers, where refs point to non-existent objects, do not
result in any bundle refs being created in the repository.
Reviewed-by: Karthik Nayak <karthik.188@gmail.com>
Reviewed-by: Patrick Steinhardt <ps@pks.im>
Signed-off-by: Xing Xin <xingxin.xx@bytedance.com>
---
bundle-uri.c | 3 +-
t/t5558-clone-bundle-uri.sh | 155 +++++++++++++++++++++++++++++++++++-
2 files changed, 152 insertions(+), 6 deletions(-)
diff --git a/bundle-uri.c b/bundle-uri.c
index 91b3319a5c1..65666a11d9c 100644
--- a/bundle-uri.c
+++ b/bundle-uri.c
@@ -400,8 +400,7 @@ static int unbundle_from_file(struct repository *r, const char *file)
refs_update_ref(get_main_ref_store(the_repository),
"fetched bundle", bundle_ref.buf, oid,
has_old ? &old_oid : NULL,
- REF_SKIP_OID_VERIFICATION,
- UPDATE_REFS_MSG_ON_ERR);
+ 0, UPDATE_REFS_MSG_ON_ERR);
}
bundle_header_release(&header);
diff --git a/t/t5558-clone-bundle-uri.sh b/t/t5558-clone-bundle-uri.sh
index 1ca5f745e73..a0895913fe9 100755
--- a/t/t5558-clone-bundle-uri.sh
+++ b/t/t5558-clone-bundle-uri.sh
@@ -3,6 +3,7 @@
test_description='test fetching bundles with --bundle-uri'
. ./test-lib.sh
+. "$TEST_DIRECTORY"/lib-bundle.sh
test_expect_success 'fail to clone from non-existent file' '
test_when_finished rm -rf test &&
@@ -19,10 +20,24 @@ test_expect_success 'fail to clone from non-bundle file' '
test_expect_success 'create bundle' '
git init clone-from &&
- git -C clone-from checkout -b topic &&
- test_commit -C clone-from A &&
- test_commit -C clone-from B &&
- git -C clone-from bundle create B.bundle topic
+ (
+ cd clone-from &&
+ git checkout -b topic &&
+
+ test_commit A &&
+ git bundle create A.bundle topic &&
+
+ test_commit B &&
+ git bundle create B.bundle topic &&
+
+ # Create a bundle with reference pointing to non-existent object.
+ commit_a=$(git rev-parse A) &&
+ commit_b=$(git rev-parse B) &&
+ sed -e "/^$/q" -e "s/$commit_a /$commit_b /" \
+ <A.bundle >bad-header.bundle &&
+ convert_bundle_to_pack \
+ <A.bundle >>bad-header.bundle
+ )
'
test_expect_success 'clone with path bundle' '
@@ -33,6 +48,16 @@ test_expect_success 'clone with path bundle' '
test_cmp expect actual
'
+test_expect_success 'clone with bundle that has bad header' '
+ # Write bundle ref fails, but clone can still proceed.
+ git clone --bundle-uri="clone-from/bad-header.bundle" \
+ clone-from clone-bad-header 2>err &&
+ commit_b=$(git -C clone-from rev-parse B) &&
+ test_grep "trying to write ref '\''refs/bundles/topic'\'' with nonexistent object $commit_b" err &&
+ git -C clone-bad-header for-each-ref --format="%(refname)" >refs &&
+ test_grep ! "refs/bundles/" refs
+'
+
test_expect_success 'clone with path bundle and non-default hash' '
test_when_finished "rm -rf clone-path-non-default-hash" &&
GIT_DEFAULT_HASH=sha256 git clone --bundle-uri="clone-from/B.bundle" \
@@ -259,6 +284,128 @@ test_expect_success 'clone bundle list (file, any mode, all failures)' '
! grep "refs/bundles/" refs
'
+test_expect_success 'negotiation: bundle with part of wanted commits' '
+ test_when_finished "rm -f trace*.txt" &&
+ GIT_TRACE_PACKET="$(pwd)/trace-packet.txt" \
+ git clone --no-local --bundle-uri="clone-from/A.bundle" \
+ clone-from nego-bundle-part &&
+ git -C nego-bundle-part for-each-ref --format="%(refname)" >refs &&
+ grep "refs/bundles/" refs >actual &&
+ test_write_lines refs/bundles/topic >expect &&
+ test_cmp expect actual &&
+ # Ensure that refs/bundles/topic are sent as "have".
+ tip=$(git -C clone-from rev-parse A) &&
+ test_grep "clone> have $tip" trace-packet.txt
+'
+
+test_expect_success 'negotiation: bundle with all wanted commits' '
+ test_when_finished "rm -f trace*.txt" &&
+ GIT_TRACE_PACKET="$(pwd)/trace-packet.txt" \
+ git clone --no-local --single-branch --branch=topic --no-tags \
+ --bundle-uri="clone-from/B.bundle" \
+ clone-from nego-bundle-all &&
+ git -C nego-bundle-all for-each-ref --format="%(refname)" >refs &&
+ grep "refs/bundles/" refs >actual &&
+ test_write_lines refs/bundles/topic >expect &&
+ test_cmp expect actual &&
+ # We already have all needed commits so no "want" needed.
+ test_grep ! "clone> want " trace-packet.txt
+'
+
+test_expect_success 'negotiation: bundle list (no heuristic)' '
+ test_when_finished "rm -f trace*.txt" &&
+ cat >bundle-list <<-EOF &&
+ [bundle]
+ version = 1
+ mode = all
+
+ [bundle "bundle-1"]
+ uri = file://$(pwd)/clone-from/bundle-1.bundle
+
+ [bundle "bundle-2"]
+ uri = file://$(pwd)/clone-from/bundle-2.bundle
+ EOF
+
+ GIT_TRACE_PACKET="$(pwd)/trace-packet.txt" \
+ git clone --no-local --bundle-uri="file://$(pwd)/bundle-list" \
+ clone-from nego-bundle-list-no-heuristic &&
+
+ git -C nego-bundle-list-no-heuristic for-each-ref --format="%(refname)" >refs &&
+ grep "refs/bundles/" refs >actual &&
+ cat >expect <<-\EOF &&
+ refs/bundles/base
+ refs/bundles/left
+ EOF
+ test_cmp expect actual &&
+ tip=$(git -C nego-bundle-list-no-heuristic rev-parse refs/bundles/left) &&
+ test_grep "clone> have $tip" trace-packet.txt
+'
+
+test_expect_success 'negotiation: bundle list (creationToken)' '
+ test_when_finished "rm -f trace*.txt" &&
+ cat >bundle-list <<-EOF &&
+ [bundle]
+ version = 1
+ mode = all
+ heuristic = creationToken
+
+ [bundle "bundle-1"]
+ uri = file://$(pwd)/clone-from/bundle-1.bundle
+ creationToken = 1
+
+ [bundle "bundle-2"]
+ uri = file://$(pwd)/clone-from/bundle-2.bundle
+ creationToken = 2
+ EOF
+
+ GIT_TRACE_PACKET="$(pwd)/trace-packet.txt" \
+ git clone --no-local --bundle-uri="file://$(pwd)/bundle-list" \
+ clone-from nego-bundle-list-heuristic &&
+
+ git -C nego-bundle-list-heuristic for-each-ref --format="%(refname)" >refs &&
+ grep "refs/bundles/" refs >actual &&
+ cat >expect <<-\EOF &&
+ refs/bundles/base
+ refs/bundles/left
+ EOF
+ test_cmp expect actual &&
+ tip=$(git -C nego-bundle-list-heuristic rev-parse refs/bundles/left) &&
+ test_grep "clone> have $tip" trace-packet.txt
+'
+
+test_expect_success 'negotiation: bundle list with all wanted commits' '
+ test_when_finished "rm -f trace*.txt" &&
+ cat >bundle-list <<-EOF &&
+ [bundle]
+ version = 1
+ mode = all
+ heuristic = creationToken
+
+ [bundle "bundle-1"]
+ uri = file://$(pwd)/clone-from/bundle-1.bundle
+ creationToken = 1
+
+ [bundle "bundle-2"]
+ uri = file://$(pwd)/clone-from/bundle-2.bundle
+ creationToken = 2
+ EOF
+
+ GIT_TRACE_PACKET="$(pwd)/trace-packet.txt" \
+ git clone --no-local --single-branch --branch=left --no-tags \
+ --bundle-uri="file://$(pwd)/bundle-list" \
+ clone-from nego-bundle-list-all &&
+
+ git -C nego-bundle-list-all for-each-ref --format="%(refname)" >refs &&
+ grep "refs/bundles/" refs >actual &&
+ cat >expect <<-\EOF &&
+ refs/bundles/base
+ refs/bundles/left
+ EOF
+ test_cmp expect actual &&
+ # We already have all needed commits so no "want" needed.
+ test_grep ! "clone> want " trace-packet.txt
+'
+
#########################################################################
# HTTP tests begin here
--
gitgitgadget
next prev parent reply other threads:[~2024-06-19 4:07 UTC|newest]
Thread overview: 66+ messages / expand[flat|nested] mbox.gz Atom feed top
2024-05-15 3:01 [PATCH] bundle-uri: refresh packed_git if unbundle succeed blanet via GitGitGadget
2024-05-17 5:00 ` Patrick Steinhardt
2024-05-17 16:14 ` Junio C Hamano
2024-05-20 11:48 ` Xing Xin
2024-05-20 17:19 ` Junio C Hamano
2024-05-27 16:04 ` Xing Xin
2024-05-20 9:41 ` Xing Xin
2024-05-17 7:36 ` Karthik Nayak
2024-05-20 10:19 ` Xing Xin
2024-05-20 12:36 ` [PATCH v2] bundle-uri: verify oid before writing refs blanet via GitGitGadget
2024-05-21 15:41 ` Karthik Nayak
2024-05-27 15:41 ` [PATCH v3 0/4] object checking related additions and fixes for bundles in fetches blanet via GitGitGadget
2024-05-27 15:41 ` [PATCH v3 1/4] bundle-uri: verify oid before writing refs Xing Xin via GitGitGadget
2024-05-28 11:55 ` Patrick Steinhardt
2024-05-30 8:32 ` Xing Xin
2024-05-27 15:41 ` [PATCH v3 2/4] unbundle: introduce unbundle_fsck_flags for fsckobjects handling Xing Xin via GitGitGadget
2024-05-28 12:03 ` Patrick Steinhardt
2024-05-29 18:12 ` Xing Xin
2024-05-30 4:38 ` Patrick Steinhardt
2024-05-30 8:46 ` Xing Xin
2024-05-27 15:41 ` [PATCH v3 3/4] fetch-pack: expose fsckObjects configuration logic Xing Xin via GitGitGadget
2024-05-28 12:03 ` Patrick Steinhardt
2024-05-28 17:10 ` Junio C Hamano
2024-05-28 17:24 ` Junio C Hamano
2024-05-29 5:52 ` Patrick Steinhardt
2024-05-30 8:48 ` Xing Xin
2024-05-29 5:52 ` Patrick Steinhardt
2024-05-27 15:41 ` [PATCH v3 4/4] unbundle: introduce new option UNBUNDLE_FSCK_FOLLOW_FETCH Xing Xin via GitGitGadget
2024-05-28 12:05 ` Patrick Steinhardt
2024-05-30 8:54 ` Xing Xin
2024-05-30 8:21 ` [PATCH v4 0/4] object checking related additions and fixes for bundles in fetches blanet via GitGitGadget
2024-05-30 8:21 ` [PATCH v4 1/4] bundle-uri: verify oid before writing refs Xing Xin via GitGitGadget
2024-05-30 8:21 ` [PATCH v4 2/4] unbundle: extend verify_bundle_flags to support fsck-objects Xing Xin via GitGitGadget
2024-06-06 12:06 ` Patrick Steinhardt
2024-06-11 6:46 ` Xing Xin
2024-05-30 8:21 ` [PATCH v4 3/4] fetch-pack: expose fsckObjects configuration logic Xing Xin via GitGitGadget
2024-05-30 8:21 ` [PATCH v4 4/4] unbundle: introduce option VERIFY_BUNDLE_FSCK_FOLLOW_FETCH Xing Xin via GitGitGadget
2024-06-06 12:06 ` Patrick Steinhardt
2024-06-11 6:46 ` Xing Xin
2024-06-11 6:42 ` [PATCH v5 0/4] object checking related additions and fixes for bundles in fetches blanet via GitGitGadget
2024-06-11 6:42 ` [PATCH v5 1/4] bundle-uri: verify oid before writing refs Xing Xin via GitGitGadget
2024-06-11 6:42 ` [PATCH v5 2/4] fetch-pack: expose fsckObjects configuration logic Xing Xin via GitGitGadget
2024-06-11 6:42 ` [PATCH v5 3/4] unbundle: extend options to support object verification Xing Xin via GitGitGadget
2024-06-11 9:11 ` Patrick Steinhardt
2024-06-11 12:47 ` Xing Xin
2024-06-11 6:42 ` [PATCH v5 4/4] unbundle: use VERIFY_BUNDLE_FSCK_FOLLOW_FETCH for fetches Xing Xin via GitGitGadget
2024-06-11 12:45 ` [PATCH v6 0/3] object checking related additions and fixes for bundles in fetches blanet via GitGitGadget
2024-06-11 12:45 ` [PATCH v6 1/3] bundle-uri: verify oid before writing refs Xing Xin via GitGitGadget
2024-06-11 19:08 ` Junio C Hamano
2024-06-17 13:53 ` Xing Xin
2024-06-11 12:45 ` [PATCH v6 2/3] fetch-pack: expose fsckObjects configuration logic Xing Xin via GitGitGadget
2024-06-11 19:20 ` Junio C Hamano
2024-06-11 12:45 ` [PATCH v6 3/3] unbundle: support object verification for fetches Xing Xin via GitGitGadget
2024-06-11 20:05 ` Junio C Hamano
2024-06-12 18:33 ` Xing Xin
2024-06-11 13:14 ` [PATCH v6 0/3] object checking related additions and fixes for bundles in fetches Patrick Steinhardt
2024-06-17 13:55 ` [PATCH v7 " blanet via GitGitGadget
2024-06-17 13:55 ` [PATCH v7 1/3] bundle-uri: verify oid before writing refs Xing Xin via GitGitGadget
2024-06-18 17:37 ` Junio C Hamano
2024-06-19 6:30 ` Xing Xin
2024-06-17 13:55 ` [PATCH v7 2/3] fetch-pack: expose fsckObjects configuration logic Xing Xin via GitGitGadget
2024-06-17 13:55 ` [PATCH v7 3/3] unbundle: extend object verification for fetches Xing Xin via GitGitGadget
2024-06-19 4:07 ` [PATCH v8 0/3] object checking related additions and fixes for bundles in fetches blanet via GitGitGadget
2024-06-19 4:07 ` Xing Xin via GitGitGadget [this message]
2024-06-19 4:07 ` [PATCH v8 2/3] fetch-pack: expose fsckObjects configuration logic Xing Xin via GitGitGadget
2024-06-19 4:07 ` [PATCH v8 3/3] unbundle: extend object verification for fetches Xing Xin via GitGitGadget
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=d8fbde2dcd49d2a2a37e35f456830c51709cd743.1718770053.git.gitgitgadget@gmail.com \
--to=gitgitgadget@gmail.com \
--cc=bupt_xingxin@163.com \
--cc=git@vger.kernel.org \
--cc=karthik.188@gmail.com \
--cc=ps@pks.im \
--cc=xingxin.xx@bytedance.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).