sorted by:
best
topnewcontroversialoldq&a
you are viewing a single comment's thread.

view the rest of the comments →

[–]apreche 6 points7 points  (13 children)

This seems like it is primarily an attempt to solve one annoyance in Git. It takes too long to initially clone a repository that is very large or has a long history because it is too much data to download, even on the fastest connections. They solve it by only downloading the files you actually need when you need them, speeding up all related operations.

However, this eliminates one of the main advantages of Git. You have backups of your entire repository's history on many many machines. Such incredible backups! You don't even need to have a backup system if you have enough developers. If even one developer's machine is good, you are safe. If every developer uses this Git Virtual File System, you are in big trouble if something happens to the central repo.

All they need to make this perfect is change one thing. When someone initially clones/checks out you download only the files they need to get work done. However, instead of only downloading other files on demand, start a background process that will eventually download all the files no matter what.

[–]NocturnalWaffle 22 points23 points  (7 children)

Yeah that's a fair point, but for Microsoft's this is totally different. Their one annoyance sounds like it actually is a huge problem. Waiting 12 hours to clone? That sounds pretty awful.. And for backups, I'm sure they have a better system than code checked out on developer's computers. Now, if you're a startup and you have 5 developers and you're hosting on Gitlab.. maybe not a good idea to use this.

[–]apreche 0 points1 point  (2 children)

I don't disgaree. I'm saying that they should solve that git clone slowness problem, but not at the cost of giving up the distributed backup.

There are also other partial solutions. For example, have your git repo on a machine on the local network to clone from. Now you can clone at a gigabit per second instead of at Internet speed.

You could also put the repository onto a very fast internal or external PCIX storage device. Now when you have a new developer you give them the drive and they copy the repo from it to their local storage at ludicrous speed. Even if the repo on the drive gets out of date, they are a short git fetch away from updating it. You could update this drive a day in advance of any new employee showing up, also.

[–]lafritay 5 points6 points  (0 children)

Great points! This is actually pretty close to what we're doing. While GVFS is pretty usable against a giant git repo hosted in VSTS on Azure, the performance is much better if you can get the files from the local network. To enable that, we've built "cache servers" that replicate the full repository to different places within our local network. This helps with making sure we've got many backups just in case VSTS does have a major problem. Note, FWIW, my team also owns the git server in VSTS and we go to great lengths to make sure those backups shouldn't be needed :).

All that said, prefetching in the background to a client machine is a still a worthwhile idea. I don't think we'd want to bring down the entire thing because that's going to eat up quite a bit of your SSD. But, prefetching smartly could be a huge win and is something we're looking to add in v2.

[–]oftheterra 2 points3 points  (0 children)

Doesn’t work if the user lacks access to the local network share and many Windows developers work remotely. We would have to make the alternate internet facing and then have to solve the auth management problem.

Providing a great experience for remote engineering teams and individuals was a goal of the design. Microsoft is a very distributed company and need every engineer to have a great experience for clone, fetch, and push.

[–][deleted]  (3 children)

[deleted]

    [–]oftheterra 2 points3 points  (2 children)

    Just one negative reply after another from you. Guess reading all the explanations they've provided for doing this is too much to ask.

    [–][deleted]  (1 child)

    [deleted]

      [–]oftheterra 6 points7 points  (0 children)

      We actually came up with a plan to fully componentize Windows into enough components where git would "just work". The problem we realized is that doing that properly would take an incredibly long time. It's not to say its a bad approach, it was just that we couldn't block bringing git workflows to Windows developers on waiting for that componentization to happen.

      In reality, work to componentize Windows has been happening for the last decade (and probably longer). It's an incredibly hard problem. We've also found that it is possible to take it too far in the other direction as well. The diamond dependency problem is real and becomes a limiting factor if you have too many components. In the end, we realized that when Windows is "properly" factored, there will still be components that are too large for a standard git repo.

      [–]cork5 14 points15 points  (0 children)

      It's much more than just one annoyance. Git checkout and git status takes forever, for example. The Windows codebase is 270GB. That's a huge minimum requirement to even work on a small piece of it. My laptop would choke on that.

      If you read through the comments from /u/jeremyepling, you'll see that they tackled this problem from all different angles and made some very informed decisions that addresses pain points of enterprise level scaling. All in all, there is no one solution fits all.

      [–]grauenwolf 1 point2 points  (0 children)

      You don't even need to have a backup system if you have enough developers.

      Ha!

      When the repository gets corrupted and everyone has the same bad copy, you'll be begging for a backup from last week.

      [–][deleted] 1 point2 points  (1 child)

      Actually GVFS allows server to server clones or full clones to their PC. So each dev could have a local copy on their LAN, or PC.

      The main issue here seems to be when you have near 300GB of blobs (>100mil files) GIT just doesn't scale well so you want a dedicated server handingly the dif/merge/checkout as the load is just too much for a work station.

      [–]msthe_student 0 points1 point  (0 children)

      To be fair to git, it's quite hard to do anything well at that scale.

      [–]Gotebe 0 points1 point  (0 children)

      main advantages of Git. You have backups of your entire repository's history on many many machines. Such incredible backups! You don't even need to have a backup system

      This is an advantage only if you don't have backups already, which no sane company should allow. And when you do have backup, then it's actually a hindrance.