* git hangs with --separate-git-dir
@ 2023-05-16 17:54 Tim Walter (Visual Concepts)
2023-05-16 21:30 ` brian m. carlson
0 siblings, 1 reply; 6+ messages in thread
From: Tim Walter (Visual Concepts) @ 2023-05-16 17:54 UTC (permalink / raw)
To: git@vger.kernel.org
Windows 10 PC
large project
300GB
500,000 files
mix of binary and text assets.
cd /d D:\myproject
git init --separate-git-dir=F:\myproject.git
git add *
Git runs for a little while then hangs and does some really bad things to the OS as well.
it seems impossible to kill git.exe even with administrator priviledges.
f: drive ends up locked and any other process, such as explorer that try to look at it also hang
you cannot log out without hanging
or even restart
This then requires a hard power cycle to fix.
100% reproducable.
running git init without --separate-git-dir and git add * works fine (takes 3 hours,.. but works)
then I can move the .git dir to f: and reinit with --separate-git-dir and that works.
So this seems to only be a problem for the initial init.
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: git hangs with --separate-git-dir
2023-05-16 17:54 git hangs with --separate-git-dir Tim Walter (Visual Concepts)
@ 2023-05-16 21:30 ` brian m. carlson
2023-05-16 21:55 ` Tim Walter (Visual Concepts)
0 siblings, 1 reply; 6+ messages in thread
From: brian m. carlson @ 2023-05-16 21:30 UTC (permalink / raw)
To: Tim Walter (Visual Concepts); +Cc: git@vger.kernel.org
[-- Attachment #1: Type: text/plain, Size: 1894 bytes --]
On 2023-05-16 at 17:54:41, Tim Walter (Visual Concepts) wrote:
> Windows 10 PC
> large project
> 300GB
> 500,000 files
> mix of binary and text assets.
>
> cd /d D:\myproject
> git init --separate-git-dir=F:\myproject.git
> git add *
>
> Git runs for a little while then hangs and does some really bad things to the OS as well.
> it seems impossible to kill git.exe even with administrator priviledges.
> f: drive ends up locked and any other process, such as explorer that try to look at it also hang
> you cannot log out without hanging
> or even restart
> This then requires a hard power cycle to fix.
>
> 100% reproducable.
>
> running git init without --separate-git-dir and git add * works fine (takes 3 hours,.. but works)
> then I can move the .git dir to f: and reinit with --separate-git-dir and that works.
> So this seems to only be a problem for the initial init.
I don't use Windows so I can't verify this, but I would not expect this
problem to occur. Even hashing 500,000 files consuming 300 GB should
not hang the computer or take 3 hours.
What kind of disk is F:? Is it an external disk (USB or such, and if
so, what speed and kind)? Is it a network drive? Is it another local
disk in the computer (SSD or HDD)?
Are you using an antivirus or firewall other than the default, or any
sort of other monitoring software? What I suspect is happening here is
that you have an antivirus intercepting Git's operations and scanning
the files for viruses, making everything really slow, and then at some
point a bug occurs in the antivirus (which may have a kernel driver) and
then things hang. This is probably made worse if F: is an external
drive or network drive.
If you are, can you try to completely uninstall that software and
reboot, and then try again?
--
brian m. carlson (he/him or they/them)
Toronto, Ontario, CA
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 263 bytes --]
^ permalink raw reply [flat|nested] 6+ messages in thread
* RE: git hangs with --separate-git-dir
2023-05-16 21:30 ` brian m. carlson
@ 2023-05-16 21:55 ` Tim Walter (Visual Concepts)
2023-05-17 1:20 ` brian m. carlson
0 siblings, 1 reply; 6+ messages in thread
From: Tim Walter (Visual Concepts) @ 2023-05-16 21:55 UTC (permalink / raw)
To: brian m. carlson; +Cc: git@vger.kernel.org
---TAW replies below---
-----Original Message-----
From: brian m. carlson <sandals@crustytoothpaste.net>
Sent: Tuesday, May 16, 2023 2:31 PM
To: Tim Walter (Visual Concepts) <twalter@vcentertainment.com>
Cc: git@vger.kernel.org
Subject: Re: git hangs with --separate-git-dir
On 2023-05-16 at 17:54:41, Tim Walter (Visual Concepts) wrote:
> Windows 10 PC
> large project
> 300GB
> 500,000 files
> mix of binary and text assets.
>
> cd /d D:\myproject
> git init --separate-git-dir=F:\myproject.git
> git add *
>
> Git runs for a little while then hangs and does some really bad things to the OS as well.
> it seems impossible to kill git.exe even with administrator priviledges.
> f: drive ends up locked and any other process, such as explorer that
> try to look at it also hang you cannot log out without hanging or even
> restart This then requires a hard power cycle to fix.
>
> 100% reproducable.
>
> running git init without --separate-git-dir and git add * works fine
> (takes 3 hours,.. but works) then I can move the .git dir to f: and reinit with --separate-git-dir and that works.
> So this seems to only be a problem for the initial init.
I don't use Windows so I can't verify this, but I would not expect this problem to occur. Even hashing 500,000 files consuming 300 GB should not hang the computer or take 3 hours.
What kind of disk is F:? Is it an external disk (USB or such, and if so, what speed and kind)? Is it a network drive? Is it another local disk in the computer (SSD or HDD)?
Are you using an antivirus or firewall other than the default, or any sort of other monitoring software? What I suspect is happening here is that you have an antivirus intercepting Git's operations and scanning the files for viruses, making everything really slow, and then at some point a bug occurs in the antivirus (which may have a kernel driver) and then things hang. This is probably made worse if F: is an external drive or network drive.
If you are, can you try to completely uninstall that software and reboot, and then try again?
--
brian m. carlson (he/him or they/them)
Toronto, Ontario, CA
---TAW--- it's SSD, actually NVme, and the PC is 128GB ram 64 core very fast.
---TAW--- Both drives are local SSD drives, the working directory on D: and the repo on F:.
---TAW--- the 3 hour thing is not a problem as such (it's a little frustrating that git is single threaded, but regardless, this is not the issue I am concerned with)
---TAW--- it's only 3 hours once, on the very first git add when I create the repo
---TAW--- I can live with that, I was just giving context.
---TAW---
---TAW--- The real issue is that it does a "very bad hang and mess up of the OS" if the repo is separate from the working directory.
---TAW--- In my case it happens to be a separate drive, not sure if that is relevant, I didn't try a separate folder on the same drive.
---TAW---
---TAW--- There's anti-virus (SentinelOne) but we're very familiar with how that works, I don't think that is the issue, and on-demand scanning is disabled anyway.
---TAW--- no firewall locally, our network obviously has firewall for the entire company network, but the workstations don't have individual firewall enabled.
---TAW--- Again, I'm not complaining about the speed, (that was just for context, and can be discussed as a separate issue if needed)
---TAW--- it's the hang and break the OS in a really nasty way that I am concerned with.
---TAW---
---TAW--- It's not possible to uninstall the anti-virus SW, company policy and I don't have permissions for that level of configuration.
---TAW--- Our current solution (perforce) manages the same source files and does not have any interaction with the anti-virus
---TAW---We're careful to tell the anti-virus to ignore certain folders, and not to do on-demand scanning on those drives anyway.
---TAW--- That's one of the reasons we have separate drives (c: is protected a lot more than d: which is just the "work folders" for example.
---TAW---
---TAW--- I expected that other people would have had this issue already, but it sounds like you've never heard of it, so maybe something odd
---TAW--- about our particular configuration? But I don't know what, except that:
---TAW--- : it's a large project
---TAW--- : it contains binary as well as text files
---TAW--- : I am trying to use 2 different local drives
---TAW--- : I am using windows, sorry can't help that, we are forced to develop using windows tools.
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: git hangs with --separate-git-dir
2023-05-16 21:55 ` Tim Walter (Visual Concepts)
@ 2023-05-17 1:20 ` brian m. carlson
2023-05-17 7:11 ` Erik Cervin Edin
0 siblings, 1 reply; 6+ messages in thread
From: brian m. carlson @ 2023-05-17 1:20 UTC (permalink / raw)
To: Tim Walter (Visual Concepts); +Cc: git@vger.kernel.org
[-- Attachment #1: Type: text/plain, Size: 3629 bytes --]
On 2023-05-16 at 21:55:35, Tim Walter (Visual Concepts) wrote:
> ---TAW--- The real issue is that it does a "very bad hang and mess up of the OS" if the repo is separate from the working directory.
> ---TAW--- In my case it happens to be a separate drive, not sure if that is relevant, I didn't try a separate folder on the same drive.
Git is an unprivileged process that doesn't install any sort of kernel
drivers. It should not have any sort of capability of hanging the OS.
If the OS is hanging, that's an OS or a driver bug.
> ---TAW--- It's not possible to uninstall the anti-virus SW, company policy and I don't have permissions for that level of configuration.
> ---TAW--- Our current solution (perforce) manages the same source files and does not have any interaction with the anti-virus
> ---TAW---We're careful to tell the anti-virus to ignore certain folders, and not to do on-demand scanning on those drives anyway.
> ---TAW--- That's one of the reasons we have separate drives (c: is protected a lot more than d: which is just the "work folders" for example.
I can tell you having answered a lot of Git-related questions on
StackOverflow and been on this list a long time, as well as being the
maintainer of Git LFS, that antiviruses often have weird behaviour just
being installed that breaks things randomly, even when fully
deactivated. For example, some antiviruses inject code into every
process, and that breaks lots of things by itself. Git is an
open source project and can't anticipate the behaviour of every
antivirus, so our semi-official policy has been that you should only use
the one shipped with the OS, if any.
It may be that Perforce happens to work here because it works
differently under the hood, but I simply can't say.
You could try running the command under the Windows Subsystem for Linux,
which often avoids antiviruses, and see if that fixes the problem for
you. It may work better and be faster as well. I like Debian for this
purpose, but Ubuntu is also very popular.
It may be that it's not antivirus-related at all, but the reason I asked
you to completely remove it and restart is to eliminate that problem,
since it's very frequently the cause, especially for random hangs. I
appreciate that that's difficult in a corporate environment, but I hope
you understand where we're coming from as well.
> ---TAW--- I expected that other people would have had this issue already, but it sounds like you've never heard of it, so maybe something odd
> ---TAW--- about our particular configuration? But I don't know what, except that:
> ---TAW--- : it's a large project
> ---TAW--- : it contains binary as well as text files
> ---TAW--- : I am trying to use 2 different local drives
> ---TAW--- : I am using windows, sorry can't help that, we are forced to develop using windows tools.
I haven't seen this problem, since I only use Windows extremely
occasionally for testing a few things. I'm unable to reproduce any sort
of problem using a separate Git dir on a separate filesystem on Linux.
You can try reporting it to the Git for Windows issue tracker at
https://github.com/git-for-windows/git/issues, and maybe they can help
you, but I'm pretty sure this is not a bug in Git itself. Git for
Windows ships a lot of code besides Git itself, and it's possible that
something there is related, though, and they'll have the knowledge to
help out more. You can try searching the issue tracker to see if
somebody has reported a similar problem before.
--
brian m. carlson (he/him or they/them)
Toronto, Ontario, CA
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 263 bytes --]
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: git hangs with --separate-git-dir
2023-05-17 1:20 ` brian m. carlson
@ 2023-05-17 7:11 ` Erik Cervin Edin
2023-05-17 16:31 ` Tim Walter (Visual Concepts)
0 siblings, 1 reply; 6+ messages in thread
From: Erik Cervin Edin @ 2023-05-17 7:11 UTC (permalink / raw)
To: brian m. carlson, Tim Walter (Visual Concepts),
git@vger.kernel.org
On Wed, May 17, 2023 at 12:24 AM Tim Walter (Visual Concepts)
<twalter@vcentertainment.com> wrote:
>
> it's SSD, actually NVme, and the PC is 128GB ram 64 core very fast.
> Both drives are local SSD drives, the working directory on D: and the repo on F:.
> the 3 hour thing is not a problem as such (it's a little frustrating that git is single threaded, but regardless, this is not the issue I am concerned with)
> it's only 3 hours once, on the very first git add when I create the repo
> I can live with that, I was just giving context.
>
I've worked on large repos (100k+ files) and 3 hours sounds unreasonable to me.
That's what, 27.7 mb/s?
My guess would probably also be anti-virus or some privilege
management type software.
Maybe try using procmon while you init and monitor all the processes
to get a better understanding of what could be the problem
https://learn.microsoft.com/en-us/sysinternals/downloads/procmon
I've used that successfully in the past to figure out the source of
unexplained performance problems (spoiler: it was privilege
management)
^ permalink raw reply [flat|nested] 6+ messages in thread
* RE: git hangs with --separate-git-dir
2023-05-17 7:11 ` Erik Cervin Edin
@ 2023-05-17 16:31 ` Tim Walter (Visual Concepts)
0 siblings, 0 replies; 6+ messages in thread
From: Tim Walter (Visual Concepts) @ 2023-05-17 16:31 UTC (permalink / raw)
To: Erik Cervin Edin, brian m. carlson, git@vger.kernel.org
Ok Thanks Erik,
I don't think I'll be able to get our security team to let me uninstall the anti-virus software.
I'm going to just create the repo without --separate-git-dir and then move it after
Which does work.
As you said, git is not a system level driver etc, so it must be in some other driver.
Could be anything.
-----Original Message-----
From: Erik Cervin Edin <erik@cervined.in>
Sent: Wednesday, May 17, 2023 12:12 AM
To: brian m. carlson <sandals@crustytoothpaste.net>; Tim Walter (Visual Concepts) <twalter@vcentertainment.com>; git@vger.kernel.org
Subject: Re: git hangs with --separate-git-dir
On Wed, May 17, 2023 at 12:24 AM Tim Walter (Visual Concepts) <twalter@vcentertainment.com> wrote:
>
> it's SSD, actually NVme, and the PC is 128GB ram 64 core very fast.
> Both drives are local SSD drives, the working directory on D: and the repo on F:.
> the 3 hour thing is not a problem as such (it's a little frustrating
> that git is single threaded, but regardless, this is not the issue I
> am concerned with) it's only 3 hours once, on the very first git add when I create the repo I can live with that, I was just giving context.
>
I've worked on large repos (100k+ files) and 3 hours sounds unreasonable to me.
That's what, 27.7 mb/s?
My guess would probably also be anti-virus or some privilege management type software.
Maybe try using procmon while you init and monitor all the processes to get a better understanding of what could be the problem https://urldefense.proofpoint.com/v2/url?u=https-3A__learn.microsoft.com_en-2Dus_sysinternals_downloads_procmon&d=DwIFaQ&c=RKDswobrOGdp5vDCbl5XjxW8HqrsRSr80dGTvu3rE9Q&r=lgWugODVl_2jeaOcjK2bD5oXKuwC73DWE5o6RZdQlHI&m=AwjvVhBZKWaxRU7lXn3bc6W20k2WAhdJarbxIl2bm-tOoU9Ygcd137hEqlAcCcBz&s=a0fvR1gPsaxSHO_LLXUxWCuMNPlrqSuhpZNExpr9q7A&e=
I've used that successfully in the past to figure out the source of unexplained performance problems (spoiler: it was privilege
management)
^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2023-05-17 16:31 UTC | newest]
Thread overview: 6+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2023-05-16 17:54 git hangs with --separate-git-dir Tim Walter (Visual Concepts)
2023-05-16 21:30 ` brian m. carlson
2023-05-16 21:55 ` Tim Walter (Visual Concepts)
2023-05-17 1:20 ` brian m. carlson
2023-05-17 7:11 ` Erik Cervin Edin
2023-05-17 16:31 ` Tim Walter (Visual Concepts)
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).