From: Glen Choo <chooglen@google.com>
To: Fraser Hanson <fraser.hanson@gmail.com>, git@vger.kernel.org
Subject: Re: git fetch recursion problem
Date: Thu, 01 Jun 2023 15:40:35 -0700 [thread overview]
Message-ID: <kl6lzg5iztzw.fsf@chooglen-macbookpro.roam.corp.google.com> (raw)
In-Reply-To: <CA+3o5aPQ3BkbAn3u6qSMCkAfNgTcpjT+_ikdqAkkc69NBGybpw@mail.gmail.com>
Fraser Hanson <fraser.hanson@gmail.com> writes:
> The git trace output looks like this (some server names redacted):
> $ export GIT_TRACE=1
> $ git fetch
> 07:41:19.325652 git.c:439 trace: built-in: git fetch
> 07:41:19.330118 run-command.c:655 trace: run_command:
> GIT_DIR=.git git remote-https origin
> https://repomirror.eng.corporate.com/github-neovim/nvim-treesitter/nvim-treesitter-textobjects.git
> 07:41:19.337765 git.c:725 trace: exec:
> git-remote-https origin
> https://repomirror.eng.corporate.com/github-neovim/nvim-treesitter/nvim-treesitter-textobjects.git
> 07:41:19.338220 run-command.c:655 trace: run_command:
> git-remote-https origin
> https://repomirror.eng.corporate.com/github-neovim/nvim-treesitter/nvim-treesitter-textobjects.git
> 07:41:20.664527 run-command.c:655 trace: run_command: git -c
> fetch.negotiationAlgorithm=noop fetch origin --no-tags
> --no-write-fetch-head --recurse-submodules=no --filter=blob:none
> --stdin
> 07:41:20.679814 git.c:439 trace: built-in: git fetch
> origin --no-tags --no-write-fetch-head --recurse-submodules=no
> --filter=blob:none --stdin
> 07:41:20.681735 run-command.c:655 trace: run_command: git
> remote-https origin
> https://repomirror.eng.corporate.com/github-neovim/nvim-treesitter/nvim-treesitter-textobjects.git
> 07:41:20.692494 git.c:725 trace: exec:
> git-remote-https origin
> https://repomirror.eng.corporate.com/github-neovim/nvim-treesitter/nvim-treesitter-textobjects.git
> 07:41:20.693132 run-command.c:655 trace: run_command:
> git-remote-https origin
> https://repomirror.eng.corporate.com/github-neovim/nvim-treesitter/nvim-treesitter-textobjects.git
> 07:41:21.695677 run-command.c:655 trace: run_command: git -c
> fetch.negotiationAlgorithm=noop fetch origin --no-tags
> --no-write-fetch-head --recurse-submodules=no --filter=blob:none
> --stdin
> 07:41:21.715621 git.c:439 trace: built-in: git fetch
> origin --no-tags --no-write-fetch-head --recurse-submodules=no
> --filter=blob:none --stdin
> 07:41:21.717230 run-command.c:655 trace: run_command: git
> remote-https origin
> https://repomirror.eng.corporate.com/github-neovim/nvim-treesitter/nvim-treesitter-textobjects.git
> 07:41:21.727616 git.c:725 trace: exec:
> git-remote-https origin
> https://repomirror.eng.corporate.com/github-neovim/nvim-treesitter/nvim-treesitter-textobjects.git
> 07:41:21.728337 run-command.c:655 trace: run_command:
> git-remote-https origin
> https://repomirror.eng.corporate.com/github-neovim/nvim-treesitter/nvim-treesitter-textobjects.git
> 07:41:22.801117 run-command.c:655 trace: run_command: git -c
> fetch.negotiationAlgorithm=noop fetch origin --no-tags
> --no-write-fetch-head --recurse-submodules=no --filter=blob:none
> --stdin
> 07:41:22.816026 git.c:439 trace: built-in: git fetch
> origin --no-tags --no-write-fetch-head --recurse-submodules=no
> --filter=blob:none --stdin
> 07:41:22.817972 run-command.c:655 trace: run_command: git
> remote-https origin
> https://repomirror.eng.corporate.com/github-neovim/nvim-treesitter/nvim-treesitter-textobjects.git
> 07:41:22.828436 git.c:725 trace: exec:
> git-remote-https origin
> https://repomirror.eng.corporate.com/github-neovim/nvim-treesitter/nvim-treesitter-textobjects.git
> 07:41:22.829219 run-command.c:655 trace: run_command:
> git-remote-https origin
> https://repomirror.eng.corporate.com/github-neovim/nvim-treesitter/nvim-treesitter-textobjects.git
> 07:41:24.242220 run-command.c:655 trace: run_command: git -c
> fetch.negotiationAlgorithm=noop fetch origin --no-tags
> --no-write-fetch-head --recurse-submodules=no --filter=blob:none
> --stdin
> 07:41:24.257211 git.c:439 trace: built-in: git fetch
> origin --no-tags --no-write-fetch-head --recurse-submodules=no
> --filter=blob:none --stdin
> 07:41:24.259264 run-command.c:655 trace: run_command: git
> remote-https origin
> https://repomirror.eng.corporate.com/github-neovim/nvim-treesitter/nvim-treesitter-textobjects.git
> 07:41:24.269718 git.c:725 trace: exec:
> git-remote-https origin
> https://repomirror.eng.corporate.com/github-neovim/nvim-treesitter/nvim-treesitter-textobjects.git
> 07:41:24.270612 run-command.c:655 trace: run_command:
> git-remote-https origin
> https://repomirror.eng.corporate.com/github-neovim/nvim-treesitter/nvim-treesitter-textobjects.git
> 07:41:25.306129 run-command.c:655 trace: run_command: git -c
> fetch.negotiationAlgorithm=noop fetch origin --no-tags
> --no-write-fetch-head --recurse-submodules=no --filter=blob:none
> --stdin
> 07:41:25.320842 git.c:439 trace: built-in: git fetch
> origin --no-tags --no-write-fetch-head --recurse-submodules=no
> --filter=blob:none --stdin
> 07:41:25.322801 run-command.c:655 trace: run_command: git
> remote-https origin
> https://repomirror.eng.corporate.com/github-neovim/nvim-treesitter/nvim-treesitter-textobjects.git
> 07:41:25.332775 git.c:725 trace: exec:
> git-remote-https origin
> https://repomirror.eng.corporate.com/github-neovim/nvim-treesitter/nvim-treesitter-textobjects.git
> 07:41:25.333522 run-command.c:655 trace: run_command:
> git-remote-https origin
> https://repomirror.eng.corporate.com/github-neovim/nvim-treesitter/nvim-treesitter-textobjects.git
> 07:41:26.332580 run-command.c:655 trace: run_command: git -c
> fetch.negotiationAlgorithm=noop fetch origin --no-tags
> --no-write-fetch-head --recurse-submodules=no --filter=blob:none
> --stdin
> 07:41:26.348349 git.c:439 trace: built-in: git fetch
> origin --no-tags --no-write-fetch-head --recurse-submodules=no
> --filter=blob:none --stdin
> 07:41:26.350225 run-command.c:655 trace: run_command: git
> remote-https origin
> https://repomirror.eng.corporate.com/github-neovim/nvim-treesitter/nvim-treesitter-textobjects.git
> 07:41:26.361171 git.c:725 trace: exec:
> git-remote-https origin
> https://repomirror.eng.corporate.com/github-neovim/nvim-treesitter/nvim-treesitter-textobjects.git
> 07:41:26.361969 run-command.c:655 trace: run_command:
> git-remote-https origin
> https://repomirror.eng.corporate.com/github-neovim/nvim-treesitter/nvim-treesitter-textobjects.git
> 07:41:27.585641 run-command.c:655 trace: run_command: git -c
> fetch.negotiationAlgorithm=noop fetch origin --no-tags
> --no-write-fetch-head --recurse-submodules=no --filter=blob:none
> --stdin
> 07:41:27.600316 git.c:439 trace: built-in: git fetch
> origin --no-tags --no-write-fetch-head --recurse-submodules=no
> --filter=blob:none --stdin
> 07:41:27.602370 run-command.c:655 trace: run_command: git
> remote-https origin
> https://repomirror.eng.corporate.com/github-neovim/nvim-treesitter/nvim-treesitter-textobjects.git
> 07:41:27.611823 git.c:725 trace: exec:
> git-remote-https origin
> https://repomirror.eng.corporate.com/github-neovim/nvim-treesitter/nvim-treesitter-textobjects.git
> 07:41:27.612607 run-command.c:655 trace: run_command:
> git-remote-https origin
> https://repomirror.eng.corporate.com/github-neovim/nvim-treesitter/nvim-treesitter-textobjects.git
> 07:41:28.666477 run-command.c:655 trace: run_command: git -c
> fetch.negotiationAlgorithm=noop fetch origin --no-tags
> --no-write-fetch-head --recurse-submodules=no --filter=blob:none
> --stdin
> 07:41:28.678956 git.c:439 trace: built-in: git fetch
> origin --no-tags --no-write-fetch-head --recurse-submodules=no
> --filter=blob:none --stdin
> 07:41:28.680665 run-command.c:655 trace: run_command: git
> remote-https origin
> https://repomirror.eng.corporate.com/github-neovim/nvim-treesitter/nvim-treesitter-textobjects.git
> 07:41:28.690534 git.c:725 trace: exec:
> git-remote-https origin
> https://repomirror.eng.corporate.com/github-neovim/nvim-treesitter/nvim-treesitter-textobjects.git
> 07:41:28.691277 run-command.c:655 trace: run_command:
> git-remote-https origin
> https://repomirror.eng.corporate.com/github-neovim/nvim-treesitter/nvim-treesitter-textobjects.git
> ^C
>
>
> Random clues and experiments:
>
> A `git clone` from our corporate mirrors always succeeds. Immediately
> after a `git clone`, I am always able to `git fetch`, so far.
> Something else must cause the repository to enter a broken state,
> possibly the addition of upstream commits. I can't verify that since
> I can't complete `git fetch.`
>
> I can fix a broken repository by deleting these settings from `.git/config`:
> promisor = true
> partialclonefilter = blob:none
> After this, `git fetch` succeeds.
> Deleting just one of these settings is not sufficient, it must be both.
>
> I copied one such broken repository out of our secure environment onto
> a mac laptop that has access to both our mirror sites and to the
> internet.
> I verified that `git fetch` from our mirror is still broken on the mac.
> Then I modified .git/config to point to the https://github.com/ url
> instead of our mirror site, leaving the 'promisor' and
> 'partialclonefilter' settings in place.
> Running `git fetch` in this state succeeded.
>
> This suggests that our internal mirror site is returning something
> different from github itself.
>
>
> Here is the trace from successfully fetching from github.com:
> $ git fetch
> 07:43:30.901275 git.c:439 trace: built-in: git fetch
> 07:43:30.904188 run-command.c:655 trace: run_command:
> GIT_DIR=.git git remote-https origin
> https://github.com/nvim-treesitter/nvim-treesitter-textobjects.git
> 07:43:30.917345 git.c:725 trace: exec:
> git-remote-https origin
> https://github.com/nvim-treesitter/nvim-treesitter-textobjects.git
> 07:43:30.918560 run-command.c:655 trace: run_command:
> git-remote-https origin
> https://github.com/nvim-treesitter/nvim-treesitter-textobjects.git
> remote: Enumerating objects: 8, done.
> remote: Counting objects: 100% (8/8), done.
> remote: Compressing objects: 100% (5/5), done.
> 07:43:31.714337 run-command.c:655 trace: run_command: git
> index-pack --stdin -v --fix-thin --promisor --pack_header=2,8
> remote: Total 8 (delta 2), reused 5 (delta 1), pack-reused 0
> 07:43:31.729793 git.c:439 trace: built-in: git
> index-pack --stdin -v --fix-thin --promisor --pack_header=2,8
> Receiving objects: 100% (8/8), 3.36 KiB | 3.36 MiB/s, done.
> Resolving deltas: 100% (2/2), completed with 1 local object.
> 07:43:31.738815 run-command.c:655 trace: run_command: git
> rev-list --objects --stdin --exclude-promisor-objects --not --all
> --quiet --alternate-refs
> 07:43:31.748286 git.c:439 trace: built-in: git
> rev-list --objects --stdin --exclude-promisor-objects --not --all
> --quiet --alternate-refs
> From https://github.com/nvim-treesitter/nvim-treesitter-textobjects
> 9c5237b..95b76b9 master -> origin/master
> 9c5237b..11ae4c7 update-readme -> origin/update-readme
> 07:43:31.757504 run-command.c:1524 run_processes_parallel:
> preparing to run up to 1 tasks
> 07:43:31.757515 run-command.c:1551 run_processes_parallel: done
> 07:43:31.757522 run-command.c:655 trace: run_command: git
> maintenance run --auto --no-quiet
> 07:43:31.765282 git.c:439 trace: built-in: git
> maintenance run --auto --no-quiet
>
> The difference starts just a few lines in, my failed fetches never see this:
> remote: Counting objects: 100% (8/8), done.
> remote: Compressing objects: 100% (5/
>
> Next I enabled both GIT_TRACE and GIT_CURL_VERBOSE to identify
> differences in the HTTP conversation between a failed fetch versus a
> successful one.
> The successful run was made by deleting the `promisor` and
> `partialclonefilter` settings from .git/config.
> I'm having some some trouble attaching these to this email, I'll have
> to include them in a follow up.
This looks a lot like the partial clone fetch is recursing infinitely.
This could happen if Git encountered a missing object, initiated a
partial clone fetch to get the missing object, then in the process of
that fetch it encountered a missing object, initiated a partial clone
fetch to get the missing object, then (etc etc etc).
I don't have good clues as to why that might be the case though.
Wild speculation: Does this reproduce if you clone without submodules,
or if you pass --no-recurse-submodules to "git fetch"? In a non-partial
clone fetch, "git fetch" may read .gitmodules to get submodule
information. Perhaps something similar is happening in the partial clone
fetch.
next prev parent reply other threads:[~2023-06-01 22:40 UTC|newest]
Thread overview: 5+ messages / expand[flat|nested] mbox.gz Atom feed top
2023-05-27 17:57 git fetch recursion problem Fraser Hanson
2023-05-30 19:11 ` Fraser Hanson
2023-06-01 22:40 ` Glen Choo [this message]
2023-06-01 22:57 ` Fraser Hanson
2023-06-19 18:40 ` Fraser Hanson
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=kl6lzg5iztzw.fsf@chooglen-macbookpro.roam.corp.google.com \
--to=chooglen@google.com \
--cc=fraser.hanson@gmail.com \
--cc=git@vger.kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).