* git hangs with --separate-git-dir @ 2023-05-16 17:54 Tim Walter (Visual Concepts) 2023-05-16 21:30 ` brian m. carlson 0 siblings, 1 reply; 6+ messages in thread From: Tim Walter (Visual Concepts) @ 2023-05-16 17:54 UTC (permalink / raw) To: git@vger.kernel.org Windows 10 PC large project 300GB 500,000 files mix of binary and text assets. cd /d D:\myproject git init --separate-git-dir=F:\myproject.git git add * Git runs for a little while then hangs and does some really bad things to the OS as well. it seems impossible to kill git.exe even with administrator priviledges. f: drive ends up locked and any other process, such as explorer that try to look at it also hang you cannot log out without hanging or even restart This then requires a hard power cycle to fix. 100% reproducable. running git init without --separate-git-dir and git add * works fine (takes 3 hours,.. but works) then I can move the .git dir to f: and reinit with --separate-git-dir and that works. So this seems to only be a problem for the initial init. ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: git hangs with --separate-git-dir 2023-05-16 17:54 git hangs with --separate-git-dir Tim Walter (Visual Concepts) @ 2023-05-16 21:30 ` brian m. carlson 2023-05-16 21:55 ` Tim Walter (Visual Concepts) 0 siblings, 1 reply; 6+ messages in thread From: brian m. carlson @ 2023-05-16 21:30 UTC (permalink / raw) To: Tim Walter (Visual Concepts); +Cc: git@vger.kernel.org [-- Attachment #1: Type: text/plain, Size: 1894 bytes --] On 2023-05-16 at 17:54:41, Tim Walter (Visual Concepts) wrote: > Windows 10 PC > large project > 300GB > 500,000 files > mix of binary and text assets. > > cd /d D:\myproject > git init --separate-git-dir=F:\myproject.git > git add * > > Git runs for a little while then hangs and does some really bad things to the OS as well. > it seems impossible to kill git.exe even with administrator priviledges. > f: drive ends up locked and any other process, such as explorer that try to look at it also hang > you cannot log out without hanging > or even restart > This then requires a hard power cycle to fix. > > 100% reproducable. > > running git init without --separate-git-dir and git add * works fine (takes 3 hours,.. but works) > then I can move the .git dir to f: and reinit with --separate-git-dir and that works. > So this seems to only be a problem for the initial init. I don't use Windows so I can't verify this, but I would not expect this problem to occur. Even hashing 500,000 files consuming 300 GB should not hang the computer or take 3 hours. What kind of disk is F:? Is it an external disk (USB or such, and if so, what speed and kind)? Is it a network drive? Is it another local disk in the computer (SSD or HDD)? Are you using an antivirus or firewall other than the default, or any sort of other monitoring software? What I suspect is happening here is that you have an antivirus intercepting Git's operations and scanning the files for viruses, making everything really slow, and then at some point a bug occurs in the antivirus (which may have a kernel driver) and then things hang. This is probably made worse if F: is an external drive or network drive. If you are, can you try to completely uninstall that software and reboot, and then try again? -- brian m. carlson (he/him or they/them) Toronto, Ontario, CA [-- Attachment #2: signature.asc --] [-- Type: application/pgp-signature, Size: 263 bytes --] ^ permalink raw reply [flat|nested] 6+ messages in thread
* RE: git hangs with --separate-git-dir 2023-05-16 21:30 ` brian m. carlson @ 2023-05-16 21:55 ` Tim Walter (Visual Concepts) 2023-05-17 1:20 ` brian m. carlson 0 siblings, 1 reply; 6+ messages in thread From: Tim Walter (Visual Concepts) @ 2023-05-16 21:55 UTC (permalink / raw) To: brian m. carlson; +Cc: git@vger.kernel.org ---TAW replies below--- -----Original Message----- From: brian m. carlson <sandals@crustytoothpaste.net> Sent: Tuesday, May 16, 2023 2:31 PM To: Tim Walter (Visual Concepts) <twalter@vcentertainment.com> Cc: git@vger.kernel.org Subject: Re: git hangs with --separate-git-dir On 2023-05-16 at 17:54:41, Tim Walter (Visual Concepts) wrote: > Windows 10 PC > large project > 300GB > 500,000 files > mix of binary and text assets. > > cd /d D:\myproject > git init --separate-git-dir=F:\myproject.git > git add * > > Git runs for a little while then hangs and does some really bad things to the OS as well. > it seems impossible to kill git.exe even with administrator priviledges. > f: drive ends up locked and any other process, such as explorer that > try to look at it also hang you cannot log out without hanging or even > restart This then requires a hard power cycle to fix. > > 100% reproducable. > > running git init without --separate-git-dir and git add * works fine > (takes 3 hours,.. but works) then I can move the .git dir to f: and reinit with --separate-git-dir and that works. > So this seems to only be a problem for the initial init. I don't use Windows so I can't verify this, but I would not expect this problem to occur. Even hashing 500,000 files consuming 300 GB should not hang the computer or take 3 hours. What kind of disk is F:? Is it an external disk (USB or such, and if so, what speed and kind)? Is it a network drive? Is it another local disk in the computer (SSD or HDD)? Are you using an antivirus or firewall other than the default, or any sort of other monitoring software? What I suspect is happening here is that you have an antivirus intercepting Git's operations and scanning the files for viruses, making everything really slow, and then at some point a bug occurs in the antivirus (which may have a kernel driver) and then things hang. This is probably made worse if F: is an external drive or network drive. If you are, can you try to completely uninstall that software and reboot, and then try again? -- brian m. carlson (he/him or they/them) Toronto, Ontario, CA ---TAW--- it's SSD, actually NVme, and the PC is 128GB ram 64 core very fast. ---TAW--- Both drives are local SSD drives, the working directory on D: and the repo on F:. ---TAW--- the 3 hour thing is not a problem as such (it's a little frustrating that git is single threaded, but regardless, this is not the issue I am concerned with) ---TAW--- it's only 3 hours once, on the very first git add when I create the repo ---TAW--- I can live with that, I was just giving context. ---TAW--- ---TAW--- The real issue is that it does a "very bad hang and mess up of the OS" if the repo is separate from the working directory. ---TAW--- In my case it happens to be a separate drive, not sure if that is relevant, I didn't try a separate folder on the same drive. ---TAW--- ---TAW--- There's anti-virus (SentinelOne) but we're very familiar with how that works, I don't think that is the issue, and on-demand scanning is disabled anyway. ---TAW--- no firewall locally, our network obviously has firewall for the entire company network, but the workstations don't have individual firewall enabled. ---TAW--- Again, I'm not complaining about the speed, (that was just for context, and can be discussed as a separate issue if needed) ---TAW--- it's the hang and break the OS in a really nasty way that I am concerned with. ---TAW--- ---TAW--- It's not possible to uninstall the anti-virus SW, company policy and I don't have permissions for that level of configuration. ---TAW--- Our current solution (perforce) manages the same source files and does not have any interaction with the anti-virus ---TAW---We're careful to tell the anti-virus to ignore certain folders, and not to do on-demand scanning on those drives anyway. ---TAW--- That's one of the reasons we have separate drives (c: is protected a lot more than d: which is just the "work folders" for example. ---TAW--- ---TAW--- I expected that other people would have had this issue already, but it sounds like you've never heard of it, so maybe something odd ---TAW--- about our particular configuration? But I don't know what, except that: ---TAW--- : it's a large project ---TAW--- : it contains binary as well as text files ---TAW--- : I am trying to use 2 different local drives ---TAW--- : I am using windows, sorry can't help that, we are forced to develop using windows tools. ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: git hangs with --separate-git-dir 2023-05-16 21:55 ` Tim Walter (Visual Concepts) @ 2023-05-17 1:20 ` brian m. carlson 2023-05-17 7:11 ` Erik Cervin Edin 0 siblings, 1 reply; 6+ messages in thread From: brian m. carlson @ 2023-05-17 1:20 UTC (permalink / raw) To: Tim Walter (Visual Concepts); +Cc: git@vger.kernel.org [-- Attachment #1: Type: text/plain, Size: 3629 bytes --] On 2023-05-16 at 21:55:35, Tim Walter (Visual Concepts) wrote: > ---TAW--- The real issue is that it does a "very bad hang and mess up of the OS" if the repo is separate from the working directory. > ---TAW--- In my case it happens to be a separate drive, not sure if that is relevant, I didn't try a separate folder on the same drive. Git is an unprivileged process that doesn't install any sort of kernel drivers. It should not have any sort of capability of hanging the OS. If the OS is hanging, that's an OS or a driver bug. > ---TAW--- It's not possible to uninstall the anti-virus SW, company policy and I don't have permissions for that level of configuration. > ---TAW--- Our current solution (perforce) manages the same source files and does not have any interaction with the anti-virus > ---TAW---We're careful to tell the anti-virus to ignore certain folders, and not to do on-demand scanning on those drives anyway. > ---TAW--- That's one of the reasons we have separate drives (c: is protected a lot more than d: which is just the "work folders" for example. I can tell you having answered a lot of Git-related questions on StackOverflow and been on this list a long time, as well as being the maintainer of Git LFS, that antiviruses often have weird behaviour just being installed that breaks things randomly, even when fully deactivated. For example, some antiviruses inject code into every process, and that breaks lots of things by itself. Git is an open source project and can't anticipate the behaviour of every antivirus, so our semi-official policy has been that you should only use the one shipped with the OS, if any. It may be that Perforce happens to work here because it works differently under the hood, but I simply can't say. You could try running the command under the Windows Subsystem for Linux, which often avoids antiviruses, and see if that fixes the problem for you. It may work better and be faster as well. I like Debian for this purpose, but Ubuntu is also very popular. It may be that it's not antivirus-related at all, but the reason I asked you to completely remove it and restart is to eliminate that problem, since it's very frequently the cause, especially for random hangs. I appreciate that that's difficult in a corporate environment, but I hope you understand where we're coming from as well. > ---TAW--- I expected that other people would have had this issue already, but it sounds like you've never heard of it, so maybe something odd > ---TAW--- about our particular configuration? But I don't know what, except that: > ---TAW--- : it's a large project > ---TAW--- : it contains binary as well as text files > ---TAW--- : I am trying to use 2 different local drives > ---TAW--- : I am using windows, sorry can't help that, we are forced to develop using windows tools. I haven't seen this problem, since I only use Windows extremely occasionally for testing a few things. I'm unable to reproduce any sort of problem using a separate Git dir on a separate filesystem on Linux. You can try reporting it to the Git for Windows issue tracker at https://github.com/git-for-windows/git/issues, and maybe they can help you, but I'm pretty sure this is not a bug in Git itself. Git for Windows ships a lot of code besides Git itself, and it's possible that something there is related, though, and they'll have the knowledge to help out more. You can try searching the issue tracker to see if somebody has reported a similar problem before. -- brian m. carlson (he/him or they/them) Toronto, Ontario, CA [-- Attachment #2: signature.asc --] [-- Type: application/pgp-signature, Size: 263 bytes --] ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: git hangs with --separate-git-dir 2023-05-17 1:20 ` brian m. carlson @ 2023-05-17 7:11 ` Erik Cervin Edin 2023-05-17 16:31 ` Tim Walter (Visual Concepts) 0 siblings, 1 reply; 6+ messages in thread From: Erik Cervin Edin @ 2023-05-17 7:11 UTC (permalink / raw) To: brian m. carlson, Tim Walter (Visual Concepts), git@vger.kernel.org On Wed, May 17, 2023 at 12:24 AM Tim Walter (Visual Concepts) <twalter@vcentertainment.com> wrote: > > it's SSD, actually NVme, and the PC is 128GB ram 64 core very fast. > Both drives are local SSD drives, the working directory on D: and the repo on F:. > the 3 hour thing is not a problem as such (it's a little frustrating that git is single threaded, but regardless, this is not the issue I am concerned with) > it's only 3 hours once, on the very first git add when I create the repo > I can live with that, I was just giving context. > I've worked on large repos (100k+ files) and 3 hours sounds unreasonable to me. That's what, 27.7 mb/s? My guess would probably also be anti-virus or some privilege management type software. Maybe try using procmon while you init and monitor all the processes to get a better understanding of what could be the problem https://learn.microsoft.com/en-us/sysinternals/downloads/procmon I've used that successfully in the past to figure out the source of unexplained performance problems (spoiler: it was privilege management) ^ permalink raw reply [flat|nested] 6+ messages in thread
* RE: git hangs with --separate-git-dir 2023-05-17 7:11 ` Erik Cervin Edin @ 2023-05-17 16:31 ` Tim Walter (Visual Concepts) 0 siblings, 0 replies; 6+ messages in thread From: Tim Walter (Visual Concepts) @ 2023-05-17 16:31 UTC (permalink / raw) To: Erik Cervin Edin, brian m. carlson, git@vger.kernel.org Ok Thanks Erik, I don't think I'll be able to get our security team to let me uninstall the anti-virus software. I'm going to just create the repo without --separate-git-dir and then move it after Which does work. As you said, git is not a system level driver etc, so it must be in some other driver. Could be anything. -----Original Message----- From: Erik Cervin Edin <erik@cervined.in> Sent: Wednesday, May 17, 2023 12:12 AM To: brian m. carlson <sandals@crustytoothpaste.net>; Tim Walter (Visual Concepts) <twalter@vcentertainment.com>; git@vger.kernel.org Subject: Re: git hangs with --separate-git-dir On Wed, May 17, 2023 at 12:24 AM Tim Walter (Visual Concepts) <twalter@vcentertainment.com> wrote: > > it's SSD, actually NVme, and the PC is 128GB ram 64 core very fast. > Both drives are local SSD drives, the working directory on D: and the repo on F:. > the 3 hour thing is not a problem as such (it's a little frustrating > that git is single threaded, but regardless, this is not the issue I > am concerned with) it's only 3 hours once, on the very first git add when I create the repo I can live with that, I was just giving context. > I've worked on large repos (100k+ files) and 3 hours sounds unreasonable to me. That's what, 27.7 mb/s? My guess would probably also be anti-virus or some privilege management type software. Maybe try using procmon while you init and monitor all the processes to get a better understanding of what could be the problem https://urldefense.proofpoint.com/v2/url?u=https-3A__learn.microsoft.com_en-2Dus_sysinternals_downloads_procmon&d=DwIFaQ&c=RKDswobrOGdp5vDCbl5XjxW8HqrsRSr80dGTvu3rE9Q&r=lgWugODVl_2jeaOcjK2bD5oXKuwC73DWE5o6RZdQlHI&m=AwjvVhBZKWaxRU7lXn3bc6W20k2WAhdJarbxIl2bm-tOoU9Ygcd137hEqlAcCcBz&s=a0fvR1gPsaxSHO_LLXUxWCuMNPlrqSuhpZNExpr9q7A&e= I've used that successfully in the past to figure out the source of unexplained performance problems (spoiler: it was privilege management) ^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2023-05-17 16:31 UTC | newest] Thread overview: 6+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2023-05-16 17:54 git hangs with --separate-git-dir Tim Walter (Visual Concepts) 2023-05-16 21:30 ` brian m. carlson 2023-05-16 21:55 ` Tim Walter (Visual Concepts) 2023-05-17 1:20 ` brian m. carlson 2023-05-17 7:11 ` Erik Cervin Edin 2023-05-17 16:31 ` Tim Walter (Visual Concepts)
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).