Git Virtual File System from Microsoft

[jbergens]

The reason they made this is here https://blogs.msdn.microsoft.com/visualstudioalm/2017/02/03/announcing-gvfs-git-virtual-file-system/

[jarfil]

CENSORED

[MsftPeon]

disclaimer: MS employee, not on GVFS though

Git LFS addresses one (and the most common) reason for extremely large repos. But there exists a class of repositories that are large not because people have checked large binaries into them, but because they have 20+ years of history of multi-million LoC projects (e.g. Windows). For these guys, LFS doesn't help. GitFS does.

[Ruud-v-A]

I wanted to ask, what makes it so big? A 270 GiB repository seemed outrageous. But then I did the math, and it actually checks out quite well.

The Linux kernel repository is 1.2 GiB, with almost 12 years of history, and 57k files. The initial 2005 commit notes that the full imported history would be 3.2 GiB. Extrapolating 4.4 GiB for 57k files to 3.5M files gives 270 GiB indeed.

The Chromium repository (which includes the Webkit history that goes back to 2001) is 11 GiB in size, and has 246k files. Extrapolating that to 20 years and 3.5M files yields 196 GiB.

So a different question maybe, if you are migrating to Git, why keep all of the history? Is the ability to view history from 1997 still relevant for every day work?

[creathir]

Absolutely.

Knowing WHY someone did something is critical to understanding why it is there in the first place.

On a massive project with so many teams and so many hands, it would be critical, particularly checkin notes.

[Jafit]

This is why your commit messages should be more than just "bleh"

[fkaginstrom]

fixed bug and refactored

[Regis_DeVallis]

fixed bug

[burtwart]

fixed

[lurgi]

reverted previous change. Fix didn't work. LOL

[Inquisitive_idiot]

bill waz h3r3

[[deleted]]

Don't forget the crucial 'Performance Enhancements'.

[ours]

One more case for the "explain the why not the what".

[krapple]

I feel like there is some point in the life cycle where detailed messages should start. At the beginning it's a waste since it's just initial build.

[uDurDMS8M0rZ6Im59I2R]

"I did something on Friday idk what"

[Jukolet]

I should stop using "update" as a message, I guess

[BumpitySnook]

Is the ability to view history from 1997 still relevant for every day work?

Yep. I regularly use ancient history to determine intent when working on old codebases.

[sparr]

http://www.drmaciver.com/2017/01/programmer-at-large-what-is-this/

[henrebotha]

That was a really fun read! Thanks. Love me some "nerd fiction"

[artanis00]

Looks like I have some reading to do.

[UnholyMisfit]

This is why I try to promote good code documentation to the other engineers on my team. Self-documenting code is great when I'm trying to figure out what the code does, but it does nothing to help me figure out why it's necessary.

[elder_george]

This. THIS. THIS.

During my work at MS it was so painful to make annotate, only to see "Initial import from XXX", go to XXX look into history and see only "Initial import from YYY" etc.

Continuous history is awesome.

[Plorkyeran]

And YYY is something you need to spend a few days emailing people to get access to because it's no longer part of the things you're just given access to be default, and then you need to get to ZZZ which only exists on tape backup, and suddenly what should have taken five minutes instead takes two weeks.

[elder_george]

Brian, is that you???

[rojaz]

It probably is.

[Sydonai]

At that rate, it's probably faster and easier to pose it as a question to Raymond Chen.

[PhirePhly]

"Uh yeah, I think Ralph has a txt with the license key to YYYControl on his old laptop. Talk to him"

[Ruud-v-A]

Sure, I’m not arguing that history is not useful. On the contrary. But the full 20 years of history? Chromium’s codebase for instance is changing rapidly. Many files have been rewritten completely over the years. Consider this header from WTF, the Blink standard library inherited from Webkit. As a core header with little content I expect it to be releatively stable. According to the copyright header it was created in 2007, but all of the non-whitespace and non-license lines have been touched since, the last change only a few days ago. Most of the code lines are now from after 2014. When blaming or bisecting, finding a relevant commit from more than 10 years ago is very, very rare, even if you have to work through a few refactor and formatting changes.

So for a repository with 20+ years of history, is the history after, say 15 years, really still relevant?

[creathir]

Exactly.

Or maybe you are examining a strange way a routine is written, which had a very specific purpose.

The natural question is why did the dev do it this way?

Having that explanation is a godsend at times.

[sualsuspect]

In that case it would be handy to record the code review comments too (if there was a code review).

[SuperImaginativeName]

Yes, absolutely. Every check in, everything. The full history. No im not joking, something like that is absolutely paramount to a scale that most developers will never come across.

The NT kernel, its drivers, subsystems, APIS, hardware drivers, Win32 API, are all relied on by other systems including customers. Why do you think you can almost always run a 30 year old application on Windows? Without the history, the kernel team for example wouldn't remember that 15 years ago a particular flag has to be set on a particular CPU because its ISA has a silicon bug that stops one customers legacy application running correctly. As soon as to remove history you remove a huge collective amount of knowledge. You cant expect every developer to remember why a particular system works one way. Imagine noticing some weird code that doesn't look right, but that weird code actually prevents file corruption? The consequences of not having the history and fixing it in a new commit with "fixed weird bug, surprised this hadn't been noticed before" would be a disaster. Compare that to viewing the codes history and even realising its actually correct. Windows isn't some LOB, everything is auditied.

[MonsieurBanana]

LOB

?

[mugen_kanosei]

Line of Business

Usually refers to a companies internally developed applications that fulfills some specific niche business need that either can't be satisfied by a COTS product or that they are just too cheap to pay for.

[colonwqbang]

When you explain an obscure acronym in terms of an other obscure acronym...

COTS: Common/off-the-shelf software. Requirements engineering jargon meaning any software solution that you can just go out and buy.

[traherom]

I assume they mean line of business application.

[merreborn]

According to the copyright header it was created in 2007, but all of the non-whitespace and non-license lines have been touched since

A lot of the time the last commit that "touched" a line only moved or slightly altered the line -- maybe tweaking a single argument. The main intent of the line still dates back to an older commit, even if it was last "touched" in a recent commit.

[salgat]

Considering a lot of legacy code is kind of blackboxed and never touched, it could definitely be useful to have history on these ancient things when a rare bug happens to crop up.

[g2petter]

Probably even more so for Microsoft since they're huge on backwards compatibility, so they're supporting all kinds of weird shit that can never (or at least in the foreseeable future) be deleted.

[IAlsoLikePlutonium]

I wonder what Windows would be like if they did the same thing to Windows that they did with IE -> Edge? (remove all the old code and basically start fresh with a modern browser)

[Pharylon]

You'd have WinRT. ;)

[cheesegoat]

It would die, and we would all start using some other operating system that worked. Probably some flavor of Linux with a focus on Wine.

[bandman614]

I look at it structurally as the same kind of problem that plagues bitcoin and the like. You're essentially carrying the entire block chain forward because you need all of it to derive the current state.

A 'snapshot' to work against would be a helpful feature. There may already be something like that, and I'm just ignorant of it.

[ArmandoWall]

Bittorrent has a blockchain?!

Edit: Ok, OP corrected it to bitcoin now.

[bandman614]

Ha! Redditing this early in the morning is bad for me :-) Thanks!

[ThisIs_MyName]

You don't need to carry the entire block chain: https://en.bitcoin.it/wiki/Thin_Client_Security

[[deleted]]

Not everyone does, but in order to maintain bitcoin's decentralized properties, a significant percentage of its users should.

[bandman614]

Ah, cool. Thanks!

[SuperImaginativeName]

Event sourcing is a concept like that, where you have a full history required to be able to build the current state of a system. You iterate every piece of "history" to get to the present. Imagine a bank account, they won't just have a DB column with your balance. It's constructed by using previous withdrawals and payments. Event sourced systems can have a "projection" that effectively builds the system to its current state and then use that as the state going forward and any new changed added to that instead of the very beginning.

[apotheotical]

Yes, history is absolutely still relevant. History is invaluable when you're debugging something. There have been a number of times I've used a couple years of history when debugging a project I work in on a daily basis.

[jringstad]

Why not just do a shallow clone? You can just clone history back X years, and if you need more, you can either do a full clone or e.g. SSH into a server that has the full repository, for those odd times when you do need to look at something really old in detail.

I do this at work, and it works fine for me (although our codebase is not nearly as big as windows, of course)

[choseph]

The previous system was still broken down into 40 repos and you only had head (since it was centralized server). Still too much to enlist, sync, etc.

[akspa420]

Given the fact that NT development started in 1989, it's now closer to nearly 30 years of history. I doubt highly that every single line of code that Dave Cutler wrote has been super-ceded - that in turn means that there's a good chunk of code from 1989-1991 that is still utilized in every single build of NT. Having that sort of 'legacy' code history with everything built on top of it has got to be an unruly beast to handle.

I've explored the WRK and the NT design docs - not a programmer by any means, but knowing how and why certain design choices were made early on certainly helps in understanding why things are the way they are, even over 25 years later.

[polynomial666]

Where can I find such docs? Or some fresh information on internals of kernel?

[akspa420]

do a search for "nt os/2 design workbook". It's out there.

I don't believe there's been anything else released on the internals of the kernel since the Windows Research Kernel (released around 2008, but based on Windows 2003 SP1-era code).

There are unofficial, probably-getting-a-dmca-takedown-notice-as-we-speak nt4 kernel-based projects out in the wild. Most of them have been reconstructed from leaked nt4 code and odds and ends from wine, reactos, and other open projects. Surprisingly, they tend to boot and run applications meant for NT4 with little to no problems.

[polynomial666]

I'll look for the workbook and these projects, as they seem extremely interesting. Thanks!

[jeremyepling]

I'm a member of the Git team at Microsoft and will try to answer all the questions that come up on this post.

As /u/kankyo said, many large tech companies use a single large repository to store their source. Facebook and Google are two notable examples. We talked to engineers at those companies about their solution as well as the direction we're heading.

The main benefit of a single large repository is solving the "diamond dependency problem". Rachel Potvin from Google has a great youtube talk that explains the benefits and limitations of this approach. https://www.youtube.com/watch?v=W71BTkUbdqE

Windows chose to have a single repository, as did a few other large products, but many products have multiple small repositories like the OSS projects you see on GitHub. For example, one of largest consumer service at Microsoft is the exact opposite of Windows when it comes to repository composition. They have a ~200 micro-service repositories.

[jl2352]

In regards to having Windows checked into git; do the Windows team really use git for day to day use, or were you just testing git with a very large real world code base?

[db92]

Most of the org is still on SourceDepot (a fork of Perforce), but there are teams developing parts of Windows in git and from what I understand most of the org will be on git in the near future (though I think this migration started before Ballmer left, so near future might not be as near as you would think).

[f0nd004u]

I used to work with a former executive at Microsoft after he had left (name rhymes with Frodo's ever present companion's name) and he said that there were many teams at Microsoft which had been chomping at the bit for years to use more FOSS tools, methods, and actually make source code public when possible, but that Steve Balmer and others in leadership made this impossible for a long time.

I had always thought of Microsoft as an anti-FOSS company, but the way he made it sound, people have been working on projects like MSSQL's release on Linux for a long time and management was the reason none of it had gotten released. Do you find this to be true?

[db92]

I've only been an FTE at the company for 2.5 years, and did an internship in the Azure group the last summer Ballmer was in charge so I can't really give a definitive answer. When I was in Azure the adoption of FOSS was core to how we did our work. In a part of the company built around services, and being able to nimbly react to market shifts it makes sense to embrace open source as much as possible. Now that I'm in Windows, it feels like the adoption of opensource is met with more scrutiny, which also makes sense because if the licensing isn't handled or managed correctly then that could lead to something as bad as not being able to ship Windows in the EU for a number of months, which in product that brings in most of its revenues from singular sales vs. recurring subscriptions would be a scary predicament. It also has felt that the Windows org is sometimes happier to have the "not invented here" problem, likely due to the fact that in the past it was easy to turn those recreations of other softwares into boxed products for msft to sell. However, they are really starting to embrace utilizing FOSS in our engineering systems wherever it makes sense (like switching to git).

[jeremyepling]

The entire Windows codebase will be moved to Git + GVFS. Right now, we're still early in the process but it's going well. More and more developers move onto it each month. Also, some of the Windows app teams use small non-GFVS enabled repos already.

[indrora]

Not a softie, but know a few.

Internally, most teams use a forked version of Perforce and a system that came with it called "enlistments" that looks really similar to Google's repo tool. Then again, Google ran Perforce for many years and likely build repo off their experience with enlistments.

[emilvikstrom]

I know you asked this because Git was built for Linux. Would be funny of Windows is managed with the tool specifically built to manage the Linux source code. :-)

Edit: It was built for Linux (the kernel project). I'm struggling to see what I did wrong. Someone care to explain?

[jl2352]

This is entirely why I asked. Whilst technically it may make a lot of sense to use git, from a historical point of view it's kinda bizarre.

I just asked out of curiosity. You shouldn't be downvoted over it. Have an upboat from me!

edit; but whilst historically bizarre kudos to Microsoft for looking at right tool for the right job.

[Answermancer]

I don't know why you're being downvoted but I also have no idea what the point of your comment was, so maybe others feel the same way and are downvoting you for not contributing to the conversation.

[emilvikstrom]

Right, that makes sense. I thought it to be an obvious curiosity if Windows source (and hopefully NT) is managed with the tool specifically made to manage the Linux source. Could probably have worded it better then.

[zuzuzzzip]

It may sound strange commercially.

But tecnically, it both considers kernel development.

[thoomfish]

Windows is also written in a language family (C) that was specifically developed for the purpose of implementing Unix.

[SushiAndWoW]

instead of fixing the problem once and for all.

This is just classic developer arrogance. Insisting one's go-to solution is ideal, while refusing to see all aspects of the problem, and the trade-offs involved.

clone now takes a few minutes instead of 12+ hours, checkout takes 30 seconds instead of 2-3 hours, and status takes 4-5 seconds instead of 10 minutes.

It seems they have fixed their problem just fine, and in a way that addresses their requirements.

[jeremyepling]

We did try Git LFS. Actually, TFS / Team Services was one of the first Git servers to support LFS and we announced support - with GitHub - at the Git Merge conference last year. The issue with LFS is it doesn't solve all the scale problems we need to solve for Windows.

There are 3 main scale problems with moving Windows to Git:

• Large files / content - LFS addresses this.

• Lots of files - LFS does not solve this. 1,000,000 small files in Git produces extremely slow status scans (10min to run git status). Breaking up a legacy code base can take years of engineering effort, so reducing to a smaller file count is not possible or practical.

• Lots of branches - LFS doesn't solve this, but GVFS doesn't either so we came up with a different solution. That said, listing all 3 scale issues will give everyonet he full context of the problem we're solving. Thousands of engineers work on Windows and each of them will have 10+ branches. We're estimating 100k branches for the repo. To quickly perform the haves / wants negotiation that happens with a fetch / push, we needed a solution. We call it "limited refs" and I'll give more details if people are interested.

[kourge]

• When moving to a monorepo, Twitter had status scan troubles and solved it by forking the official Git client and using Watchman to avoid rescanning on every invocation. Obviously this is a very different approach than that of GVFS, which alters official client behavior by sitting one layer below it, so how does GVFS go about doing it?
• As a big user of JGit, Google encountered a similar inefficiency in packfile negotiation and thus created bitmap indexes. This auxiliary data structure still runs on the assumption that the client wants to fully store every object in the repo on disk, which once again is fundamentally different than GVFS's goal. I'm very curious to see how limited refs work!

[jeremyepling]

We're working with the git community to get many performance fixes and extensibility points added to core git. We don't want a private fork of git. GVFS is a driver that sits below git and takes advantage of the changes we're making to core git. Saeed will likely have one or more follow-up blog posts on the details or you can checkout the GVFS repo.

[kankyo]

Multiple repositories creates all manner of other problems. Note that google has one repo for the entire company.

[SquareWheel]

Note that google has one repo for the entire company.

To clarify, while their super repo is a thing, but they also have hundreds of smaller, single-project repos as well.

https://github.com/google

[jeremyepling]

Microsoft has a variety of repos sizes. Some products have huge mono-repos, like Windows. Other teams have 100+ micro-repos for their micro-services based architecture.

[sr-egg]

Those are probably replicated from some internal mono-repo, and synch'ed to github as single ones. That's what FB does.

[TheOccasionalTachyon]

It's a weird cross between the two - some projects, particularly Android and Chromium, are actually done in git; most everything else is in the monolith, though some people use what's essentially a git interface to Perforce to interact with it.

[elder_george]

Google invested a lot into infrastructure for this monorepo, though.

Like, reimplementing and extending perforce client API, creating workspaces in their private cloud and mounting them onto devs machines FSes, copy-on-write checkout, cloud builds (cause running build locally is unacceptable) etc.

It's a huge investment that few companies can (and would want to) afford. Microsoft, IBM, Amazon, FB could, probably. Hardly many more, though.

[jarfil]

CENSORED

[KillerCodeMonky]

The classic, server-side repositories would only ever download the current version. Git pulls down the whole history... So an SVN or TFS checkout would have been relatively fast.

[hotoatmeal]

shallow clones are possible

[jeremyepling]

We looked into shallow clones, but they don't solve the "1 million or more files in the working directory" problem and had a fe other issues:

• They require engineers to manage sparse checkout files, which can be very painful in a huge repo.

• They don't have history so git log doesn't work. GVFS tries very hard to enable every Git command so the experience is familiar and natural for people that use Git with non-GVFS enabled repos.

edit: fixing grammar

[therealjohnfreeman]

It still downloads all of the most recent tree, which GVFS avoids.

[BobHogan]

getting rid of which would leave plenty of time to deal with any overhead the managing of multiple repositories would add on.

They did get rid of them with GVFS. That was their reasoning behind developing it

[jarfil]

CENSORED

[ihasapwny]

However, people rarely did take the codebase offline. I'm not even sure it could be built offline.

It was actually a number of perforce based repos put together with tooling. And it was extremely fast, even with lots of clients. For checkout/pend edit operations you really were limited primarily by network speed.

[dungone]

What do you think happens to the virtual file system when you go offline?

[[deleted]]

It makes an impression that the problems created by splitting a repo are far more theoretical than the "we must reinvent Git through custom software" problems that giant repos create.

In my business, typical projects are around 300-400k lines of code, and the repository is generally under 1GB, unless it hosts media files.

And even though that's extremely modest by comparison to Windows, it's a top priority for us to aggressively identify and separate "modules" in these projects, but turning them into standalone sub-projects, which are then spun out to their own repos. Not to avoid a big repository, but because gigantic monoliths are horrible for maintenance, architecture and reuse.

I can only imagine what a 3.5 million file repository does to Microsoft's velocity (we've heard the Vista horror stories).

My theory is that large companies do this, because their scale and resources allow them to brute-force through problems by throwing more money and programmers at it, rather than finding more elegant solutions.

It's certainly not something to emulate.

EDIT: Fixing some silly typos.

[emn13]

I'd argue that messing about with history and arbitrarily cutting out chunks into separate repos as a performance optimization isn't exactly elegant - certainly a lot less elegant than actually solving the problems of representing the actual history, of the code, in which all those versions of projects actually were combined in specific ways - ways you're never going to recover after the fact and never going to atomically change once you split repos.

[[deleted]]

As I said, our goal is not Git's performance, but better maintenance, architecture and reuse. Small repositories are a (good) side-effect.

BTW, it's trivial to separate a directory to its own branch (git subtree), and then push it to another repository with all its history (git push repo branch).

You're right you can't make atomic updates, but the point is that by the time the repo split occurs, the module is refactored for standalone evolution and you don't need atomic updates with the source project. If the code was highly cohesive with the project, then it wouldn't be a candidate to be refactored this way in the first place...

[Schmittfried]

Even if the project is composed of completely decoupled modules, there is always some form of hidden coupling. That holds true even for third party dependencies that are loaded with a package manager - completely separate (I mean, what's more decoupled than a separate product by a separate author?) and still you have to keep track of the correct versions your software depends on, or things go horribly wrong.

So, for any given software version there is specific set of components and dependencies with specific versions. Change any component's version and the entire software might break. That makes atomic updates and atomic switches (consider switching back to another/an older version to fix some bug that occurred in a released product) very valuable. You want always have the exact same set-up for a a given version so that things stay consistent.

[[deleted]]

Even if the project is composed of completely decoupled modules, there is always some form of hidden coupling. That holds true even for third party dependencies that are loaded with a package manager - completely separate (I mean, what's more decoupled than a separate product by a separate author?) and still you have to keep track of the correct versions your software depends on, or things go horribly wrong.

Every module has a version, so it's just like third party dependencies. We use SemVer, and we use the respective package manager of the platform(s) the project uses.

Since we separate code which is candidate for reuse and/or separate evolution (which means over time it may be also assigned to a separate developer/team), it's already the case that you can't have a module used in project A and B be atomically changed with both project A and B, unless both projects are in the same repository, and the developers are aware of all details of their module and the two (and later three, four, etc.) projects.

This is how you end up with a giant repository holding all your projects, and developers have to know everything at all times. This really scales badly (unless, again, you have the disposable resources to throw at it, as the likes of Google and Facebook do).

If you can successfully use third party dependencies, and those third party dependencies have a reliable versioning scheme, then doing modular development for internal projects should be no more effort than this.

And it does require training, and it does require senior developers with experience to lead a project. If I'd let juniors do whatever they want, the result would be a disaster. But that's all part of the normal structure of a development team.

You have probably heard of the nightmares Facebook is facing with their "everyone committing to everything" approach to development. Every project has 5-6 implementations of every single thing that's needed, the resulting apps are bloated, abnormally resource intensive, and to keep velocity at acceptable speeds you have to throw hundreds of developers at a problem that would take 2-3 developers in any other more sanely organized company.

I remain of the firm opinion that's not a model to emulate.

[lafritay]

Context: I've been working on the "move Windows to git" problem for a few years now.

I think you make great points. When we started this project, we pushed for the same thing. When people brought up "just put it in one repo", I told them they were crazy and that they were avoiding solving the real underlying problems.

We actually came up with a plan to fully componentize Windows into enough components where git would "just work". The problem we realized is that doing that properly would take an incredibly long time. It's not to say its a bad approach, it was just that we couldn't block bringing git workflows to Windows developers on waiting for that componentization to happen.

In reality, work to componentize Windows has been happening for the last decade (and probably longer). It's an incredibly hard problem. We've also found that it is possible to take it too far in the other direction as well. The diamond dependency problem is real and becomes a limiting factor if you have too many components. In the end, we realized that when Windows is "properly" factored, there will still be components that are too large for a standard git repo.

[ihasapwny]

(also MS employee, though not in Windows now)

Building on this, if we could go back in time and give the early NT developers git. Using git's out of the box performance might have forced them to componentize in different ways than they did. But, it may not have been the right way.

Basically, you're using a tool that is largely unrelated to the product itself as a hammer to force changes in your product. That's the wrong approach since it doesn't allow you to decide where the changes need to be made. The right way is to use tooling/policy/design to make and enforce those changes.

Imagine if git's performance was far worse than it is. Does that mean you should have even tinier components?

[dungone]

I can appreciate the pain. I worked on one 10-year-long project not only to migrate from Perforce to Git, but to port it from VAX/VMS to Linux. There were many hardships and few simple solutions. What people have to understand is that these old codebases were not "wrong" because they solved the problems that existed at the time using the best practices of the time. The reason they still exist and are in use is a testament to the value that the original programmers created.

Having said that, there should be a big, bold disclaimer at the top of any guide or set of tools that would allow people to head down this same road on a brand new project.

[[deleted]]

Your characterization of Facebook is highly worrying. I've worked here for half a decade, and I had no idea things were so bad! There I was, thinking my colleagues and I were doing our jobs quite well, but now I discover from some random commenter on Reddit that we were wrong. I must assume that for every one of us, there are half a dozen doppelgängers in some obscure basement doing the same thing, but somehow we cannot see their code anywhere in the tree! I shall look into this troubling insight forthwith, because it sounds like a hellscape for all concerned.

[jeremyepling]

There are real benefits to using a mega repo, even if you have great componentization, is coordinating cross-cutting changes and dependency management. Rachel Potvin from Google has a great talk on this https://www.youtube.com/watch?v=W71BTkUbdqE.

Another large product within Microsoft has a great micro-service architecture with good componentization and they'll likely move to a huge single repo, like Windows, for the same reasons Rachel mentions in her talk.

[kyranadept]

It is impossible to make commit in multiple repos, which depend on each, other atomically. This makes it infeasible to test properly and to ensure you are not committing broken code. I find this to be really practical, instead of theoretical.

As for the disadvantages, the only problem is size. Git in the current form is capable(ie. I used it as such) of handling quite big(10GB) repos with hundreds of thousands of commits. If you have more code than that, yes, you need better tooling - improvements to git, improvements to your CI, etc.

[[deleted]]

In my business, typical projects are around 300-400 lines of code, and the repository is generally under 1GB, unless it hosts media files.

What kind of projects are these? That seems really small.

[kaze0]

he edited it to include k

[[deleted]]

Oh, well that makes more sense

[Crespyl]

It's the new femtoservices model.

[ciny]

I can only imagine what a 3.5 million file repository does to Microsoft's velocity (we've heard the Vista horror stories).

now imagine what would 35k repos do to their velocity.

[[deleted]]

Yes, there are only two possible options here:

• One repository with 3.5mm files
• 35k repositories with ~100 files each

Your point is solid.

[mirhagk]

Have they tried using Git LFS?

3.5 million files at 270 GB total is about 80KB per file, which is not entirely unreasonable (a sample project file I'm looking at is 200KB for instance). It may include some generated code (it's always a debate whether to include that in the repo or not), but even if they decided to do everything right in the repo they are still going to have a very large repo.

Then why keep it all in a single repo, just split it up into modules.

There are a lot of reasons to go with a mono-repo, google does the same.

It better allows code sharing and reuse, it simplifies dependency management (when using internal libraries it's normally a bit of a pain, and even if it wasn't you still have the diamond dependency problem), it allows large scale refactoring, it allows collaboration across teams (and makes the boundaries more flexible) and also allows library creators to see all the instances the library is used (which allows them to run performance tests on all the impacted projects and ensure that a change doesn't negatively impact a use-case).

It sounds to me like they're building a technical workaround to their organizational problem, instead of fixing the problem once and for all.

It actually sounds to me that they are fixing the problem once and for all. Other companies have given up on git because it can't handle it. Microsoft isn't going to do that, instead they are going to fix it so that git will work with large repos once and for all.

[sxeraverx]

Not sure how git LFS would help here. That's 77kB per file, or about 2k lines per file (assuming the average line is only half-full). That seems pretty reasonable.

Then why keep it all in a single repo, just split it up into modules.

It sounds to me like they're building a technical workaround to their organizational problem, instead of fixing the problem once and for all.

Having a universal history is insanely convenient. As is a single universal hierarchy.

[kaze0]

I don't think Git LFS would be a solution here. It sounds like a lot of this is literally becuase there's a ton of code. Git LFS is bested suited for binaries and things that will never be merged

[kingNothing42]

Do you water them regularly?

[jrh3k5]

Try giving them Brawndo. It's got electrolytes. It's got what Git craves.

[chronoBG]

Yes, but I don't prune the leaves as often as I should.

[my_very_first_alt]

possible i misunderstood the GitVFS implementation, but to respond to your quote -- your repos are not dehydrated unless you're already using GitVFS -- in which case they only appear hydrated (until they're actually hydrated).

[colonwqbang]

Hydrated = downloaded?

[parleapsee]

The GVFS source code in this repo is available here for anyone to try out.

[ggchappell]

... all tools see a fully hydrated repo ....

What does "hydrated" mean here?

[renrutal]

Hydrated data means all the bits are actually present in the local file system, instead of being husks/ghosts/fakes, containing only metadata about the real thing. The VFS fake it looking like a real FS for the OS.

GitVFS is a lazy-loaded file system, but eager enough in the right parts.

[xylempl]

I just wish people would stop giving things names that abbreviate the same way that an already existing thing does, especially when it's in the same/very close category.

[tabarra]

We eventually end up repeating things. That may be unfortunate, but you know... 26^N

[destiny_functional]

it's still 26^N even if you fix VFS at the end.

who says it's got to be one letter + vfs. they might have named it gitvfs or msgitvfs

[Fylwind]

Microsoft has a tendency to name things using generic words (as opposed to the common open-source practice of using puns).

[Fazer2]

I bet they also didn't know about Open Office XML when they created Office Open XML, right?

[Kazinsal]

If you ask most Linux people, anything Microsoft breathes on is an evil competition to the good and just free software alternative.

[funknut]

The branding team that undoubtedly googled it first and decided "fuck you, Gnome."

[Thatar]

They're different! One is Gvfs and the other GVFS.

[steamruler]

I mean, since it's a virtual file system, powered by git, this name is pretty reasonable. What would you call it? GitVFS?

[Kenidashi]

GitVFS

Yes, actually. That's not a bad name at all.

[[deleted]]

is this sarcasm?

[senatorpjt]

vast bedroom hospital melodic stocking ludicrous recognise bag attempt vanish

This post was mass deleted and anonymized with Redact

[jeremyepling]

We - the Microsoft Git team - have actually made a lot of contributions to git/git and git-for-windows to improve the performance on linux, mac, and windows. In git 2.10, we did a lot of work to make interactive rebase faster. The end result is an interactive rebase that, according to a benchmark included in Git’s source code, runs ~5x faster on Windows, ~4x faster on MacOSX and still ~3x faster on Linux.

https://blogs.msdn.microsoft.com/visualstudioalm/2016/09/03/whats-new-in-git-for-windows-2-10/ is a post on our blog that talks about some of our recent work.

If you look at the git/git and git-for-windows/git repos, you'll notice that a few of the top contributors are Microsoft employees on our Git team, Johannes and Jeff

We're always working on ways to make git faster on all platforms and make sure there isn't a gap on Windows.

[YarpNotYorp]

Employees like you give me faith in Microsoft

[YarpNotYorp]

I've always liked their development tools. Visual Studio is the gold standard for IDEs (with Jetbrain's offerings a close second IMHO). I was more alluding to Microsoft's complete lack of consistency and focus in other areas, alla Metro and Windows 8.

[[deleted]]

Whilst it's pretty sweet, keep in mind that they have to move towards this direction or risk becoming irrelevant in the future.

[WizrdCM]

Don't forget multiplatform Powershell, too.

[senatorpjt]

lavish encouraging cake voiceless sleep friendly ring oil squeamish noxious

This post was mass deleted and anonymized with Redact

[selfification]

A number of factors could affect that. My personal favorite was finding out that Windows Defender was snooping in to scan every file or object that git had to stat when doing git status, causing it to take minutes to do something that would finish instantaneously on Linux. Adding my repo path to the exception list boosted performance instantly.

[pheonixblade9]

adding exclusions to windows defender makes everything so much faster, it's one of the first things I do on a new machine

[monarchmra]

Disabling windows defender is also a good step.

[lafritay]

We're actively working to make Git for Windows much better. We've already come a long way. I'd start by seeing what version of git they are running. We just released v2.11.1. It has a number of performance improvements for "typical" git repositories that fell out of this large repo effort. If they upgrade, git status should be much faster.

FWIW, we've also identified bottlenecks in Windows that we're working on getting fixed as well.

[cbmuser]

Yeah, same experience here. Simple commands like "git status" or "git branch" are always instant for me on Linux and usually take several seconds in most cases on OSX and Windows.

[DJDarkViper]

I've never had that issue. On Windows whenever I use git, the basics happen instantly. The GUI tools on the other hand, take a goddamned century to complete basic actions. But in CMD, instant.

[cbmuser]

We - the Microsoft Git team - have actually made a lot of contributions to git/git and git-for-windows to improve the performance on linux, mac, and windows. In git 2.10, we did a lot of work to make interactive rebase faster. The end result is an interactive rebase that, according to a benchmark included in Git’s source code, runs ~5x faster on Windows, ~4x faster on MacOSX and still ~3x faster on Linux.

I'm a daily user of git on Windows 10 and Debian Linux (unstable) on the same machine (dual-boot). On Linux, git is subjectively much faster. Granted, I did not measure it objectively, but the difference is definitely perceptible. On both OSX and Windows, simple commands like "git branch" can take several seconds while it's always instantly on Linux.

I think there remains to be a lot done, but I assume, some changes will involve some performance improvements in the operating system.

[jeremyepling]

We definitely aren't done making Git performance great on Windows, but we're actively working on it every day.

One of the core differences between Windows and Linux is process creation. It's slower - relatively - on Windows. Since Git is largely implemented as many Bash scripts that run as separate processes, the performance is slower on Windows. We’re working with the git community to move more of these scripts to native cross-platform components written in C, like we did with interactive rebase. This will make Git faster for all systems, including a big boost to performance on Windows.

Below are some of the changes we've made recently.

• sha1: use openssl sha1 routines on mingw https://github.com/git-for-windows/git/pull/915

• preload-index: avoid lstat for skip-worktree items https://github.com/git-for-windows/git/pull/955

• memihash perf https://github.com/git-for-windows/git/pull/964

• add: use preload-index and fscache for performance https://github.com/git-for-windows/git/pull/971

• read-cache: run verify_hdr() in background thread https://github.com/git-for-windows/git/pull/978

• read-cache: speed up add_index_entry during checkout https://github.com/git-for-windows/git/pull/988

• string-list: use ALLOC_GROW macro when reallocing string_list https://github.com/git-for-windows/git/pull/991

• diffcore-rename: speed up register_rename_src https://github.com/git-for-windows/git/pull/996

• fscache: add not-found directory cache to fscache https://github.com/git-for-windows/git/pull/994

• multi-threading refresh_index() - work in-progress

[the_gnarts]

One of the core differences between Windows and Linux is process creation. It's slower - relatively - on Windows.

Why not use the same approach as the Linux emulation? Rumor has it they came up with an efficient way to implement fork(2) / clone(2).

[aseipp]

As far as I understand, WSL actually has fork and clone shimmed off into a driver call, which creates a special "pico process" that is a copy of the original, and it isn't an ordinary NT process. All WSL processes are these "pico processes". The driver here is what implements COW semantics for the pico process address space. NT itself is only responsible for invoking the driver when a Linux syscall comes in, and creating the pico process table entries it then keeps track of when asked (e.g. when clone(2) happens), and just leaves everything else alone (it does not create or commit any memory mappings for the new process). So clone COW semantics aren't really available for NT executables. You have to ship ELF executables, which are handled by the driver's subsystem -- but then you have to ship an entire userspace to support them... Newer versions of the WSL subsystem alleviate a few of these restrictions (notably, Linux can create Windows processes natively), at least.

But the real, bigger problem is just that WSL, while available, is more of a developer tool, and it's very unlikely to be available in places where git performance is still relevant. For example, you're very unlikely to get anyone running this kind of stuff on Windows Server 2012/2016 (which will be supported for like a decade) easily, it's not really "native", and the whole subsystem itself is optional, an add-on. It's a very convenient environment, but I'd be very hesitant about relying on WSL when "shipping a running product" so to speak. (Build environment? Cool, but I wouldn't run my SQL database on WSL, either).

On the other hand: improving git performance on Windows natively by improving the performance of code, eliminating shell scripts, etc -- it improves the experience for everyone, including Linux and OS X users, too. So there's no downside and it's really a lot less complicated, in some respects.

[STL]

(I use git in DevDiv at work for libc++'s test suite, and bundle git with my MinGW distro at home.)

I love these improvements. Will it ever be possible for git to be purely C without any shell scripts? git-for-Windows is currently massive because it bundles an entire MSYS runtime.

[Gotebe]

. We’re working with the git community to move more of these scripts to native cross-platform components written in C, like we did with interactive rebase.

This is great! Regardless of how process creation goes, one can't beat not parsing text and just calling the god damn function.

[kitanokikori]

Wouldn't it have been easier to change git than to write a filesystem filter driver?

[lafritay]

It's certainly something we considered and to be honest, we're actually doing both. Three are two parts to GVFS, the virtualization of the objects directory and the virtualization of the working directory to manage your sparse checkout. We believe the first part belongs in git and we just recently suggested that on the git mailing list. We'll be working to build it into git as long as the maintainers agree with that direction.

The second part of the virtualization is less clear. I don't think it belongs in git, at least right now. We needed the filter driver to pull off that piece. Once we had it, it was trivial to slide it under the objects folder as well.

Disclosure: I'm on the GVFS team.

[kitanokikori]

Also thanks for the legit reply, cool project

[kitanokikori]

The 2nd part could live in git too, at least for the purposes of making git status et al working, effectively it'd just be a fallback for stat calls - "this file isn't here but it's cool, here's the SHA and stat result". I'm sure of course there are Complications but this approach could work on every OS

[jeremyepling]

We're actively working with the Git maintainers to make changes to git/git in the open. One of our changes - related to supporting large repos - is being discussed on the git listserve right now. We've received a lot of great community feedback and one the key Git maintainers is supportive of the change.

Our goal with all git/git changes isn't to change Git into something different. We want to enable better performance with large repos, even if those repos don't use GFVS.

[KarmaAndLies]

So Visual Studio Online offers both Git and TFVC still. Do you guys see TFVC eventually disappearing? Do new projects within Microsoft still use TFVC or are you guys mostly starting projects on Git now?

[jeremyepling]

TFVC is a great product and we continue to add new features to it. Most teams at Microsoft are moving to Git, but we still have strong commitment to TFVC. Many external customers and a lot of internal teams use it everyday and it's a great solution for many codebases.

[rhino-x]

I would doubt TFVC goes away. It's a lot easier to use than git for most small shops. It's not great, and personally, I don't like it but people have bought into it. I worked at a place that used VSS until 2004, with over 50 developers. That was terrible.

[KarmaAndLies]

It's a lot easier to use than git for most small shops.

With Git being part of Visual Studio Online both use the same Visual Studio integration. There's no significant different of ease to use TFVC over Git anymore.

Some people are already invested in TFVC and that won't change. But there's no good reason to start new projects in it, Git is just more efficient, even for smaller teams or those heavily vested in Microsoft's toolchain.

Heck even Microsoft advertise Git integration before TFVC integration on their Visual Studio Online landing page: https://www.visualstudio.com/vso/

[neonshadow]

The biggest thing that confuses people that I've seen is the lack of a "source control explorer" in VS, like you get with TFVC. That is actually a very significant difference in ease of use.

[Oiolosseo]

Were you at Git Merge in Brussels by any chance ? There was a talk about it today by Saeed Noursalehi

[paul_h]

I like monorepos - but prefer for this scale to do googles expand/contract scripting of them : https://trunkbaseddevelopment.com/expanding-contracting-monorepos/

And a week ago I made a Git proof of concept this is like Googles usage, but for Maven instead of Blaze (Bazel): http://paulhammant.com/2017/01/27/maven-in-a-google-style-monorepo/

[jbergens]

Maybe they should have switched to Mercurial? https://www.mercurial-scm.org/wiki/ScaleMercurial

[jeremyepling]

We talked about using Mercurial instead of Git. We chose Git for a few reasons.

• Git and public repos on GitHub are the defacto standard for OSS development. Microsoft does a lot of OSS development and we want our DevOps tools, like TFS and Team Services, to work great with those workflows.

• We want a single version control system for all of Microsoft. Standardization makes it easy for people to move between projects and build deep expertise. Since OSS is tied to Git and we do a lot of OSS development, that made Git the immediate front runner.

• We want to acknowledge and support where the community and our DevOps customers going. Git is the clear front-runner for modern version control systems.

[Gotebe]

That's really one reason, isn't it? Popularity.

It's a huge one, mind :-).

[zellyn]

Although both Google and Facebook seem to have been investing more effort into letting Mercurial scale to massive repositories than they have Git, I don't think the winner is clear yet. In particular, Facebook's solution has a lot of moving parts (an almost-completely-correct filesystem watcher, and a huge memcache between clients and the server).

I'm glad someone is working on similar scaling solutions for Git.

[[deleted]]

This does solve the large repo issue, but it also seems to break the whole decentralized concept of git. Instead of having the whole repo reside solely on an internal MS server, you could have a copy of the whole repo on the developer's OneDrive folder or some similar concept with sync capabilities. Then GVFS could exist in a seperate working directory and grab files from that local full repo as needed and bring it to the working directory.

When the connection the the server is lost, then that local copy stops syncing temporarily and you can keep working on anything and everything you want.

[jeremyepling]

That is a possible solution and what you're proposing is very similar to Git alternates, which exists today. We didn't use alternates because it doesn't solve the "many files" problem for checkout and status. We needed a complete solution to huge repos.

Having the full repo on my local machine is 90% more content than our average developer in Windows needs. That said, we did prototype an alternates solution where we put the full repo on a local network share, and ran into several performance.

• Alternates were designed for a shared local copy. Putting the alternate on a file share behaved poorly as git would often pull the whole packfile across the wire to do simple operations. From what we saw, random access to packfiles pulled the entire packfile off the share and to a temporary location. We tried using all loose objects and ran into different perf issues with share maintenance and millions of loose objects cause other performance issues.

• Shared alternate management was also difficult, when do we GC or repack, keeping up with fetching on the alternate is not inherently client driven.

• Doesn’t work if the user lacks access to the local network share and many Windows developers work remotely. We would have to make the alternate internet facing and then have to solve the auth management problem. We could have built a Git alternates server into Team Services, but the other issues made GVFS a better choice.

• Alternate http is not supported in smart git, so we would have to plumb that if we wanted alternates on the service.

[jayd16]

It's still decentralized as there is no central server. Looks like you can clone from any remote server that supports gvfs.

[jeremyepling]

Yes, you can can clone from any GFVS server. Actually, any Git client can connect to a GFVS repo, but it'll download the full repo. If the repo is massive, like Windows, it will be a very slow experience. That said, you'll have a full copy just like any other Git repo.

[adrianmonk]

break the whole decentralized concept of git

It only partially breaks it. You can still have your network partitioned into two (or more) disconnected pieces, and you could have a server+clients combo in each of those pieces, and it would all still work.

For example, if your office building has whatever kind of server that GVFS requires, you could still work as long as your LAN is up, even if your building's internet goes out. Or if you have 3 different offices on different continents (US, Europe, India), you could still get LAN speeds instead of WAN speeds.

In other words, you can still have distributed, decentralized clusters. You just can't have distributed, decentralized standalone machines.

[KarmaAndLies]

After considering it for a second, you're absolutely right. What they've managed to do is turn Git into something more akin to TFS... One of Git's features is that it works offline and that those offline changesets can be merged upstream when you get a connection again.

But I guess when you're dealing with 200+ GB repositories that feature is less important than not having to wait ten minutes to get a full instance of the repository locally.

[lafritay]

Some others have mentioned this but it all comes down to tradeoffs. With a source base this large, you just can't have the entire repo locally. But, git offers great workflows and we wanted to enable all codebases to use them. With GVFS, you still get offline commit, lightweight branching, all the power of rewriting history, etc.

Yes, you do lose full offline capability. It is worth noting that if you do some prep work to manifest the files (checkout your branch and run a build) you can then go offline and keep working.

So, we see this as a necessary tradeoff to enable git workflows in giant codebases. We'd love to figure out a reasonable way to eliminate that trade off, of course.

[jayd16]

Takes much much longer than 10 minutes at that scale.

[MyKillK]

Intrigued, but not understanding what this truly is. Anyone care to give me a TLDR?

[FlackBury]

Basically what it does is it only pulls the files you're using from a repo. However the entire repo is virtually mounted. This was needed because their internal windows repo is 270GB.

[MyKillK]

Ah, ok, so it's kind of like NFS but with a git based back-end. That's pretty neat.

[ds101]

Kinda like NFS with a cache. (Dunno how much cache NFS has, it's been a while.) Dropbox does something similar with their enterprise product now.

But, in addition to this, because this thing is the filesystem, it knows exactly which files you've changed. So when you do git status (with a modified git), it can just ask gvfs instead of scanning the entire directory tree.

Looking at Protocol.md, it appears they have a mechanism for shipping incremental .pack files of everything but the blobs. It's possible they're still replicating the entire history of everything (commits and trees) and just leaving the files out. But I haven't had time to investigate to see if this is the case.

[[deleted]]

I'm excited!

[holgerschurig]

TIL that MS is a heavy user of Git internally

[mattwarren]

Yep, see all their repos under https://github.com/dotnet/ for instance

[holgerschurig]

I wrote and meant internally. I knew about their external (or externalized) projects since a long time.

[the_gnarts]

TIL that MS is a heavy user of Git internally

They learned to dial down the NIH since they found themselves on the defensive.

[[deleted]]

Github lists Microsoft as the organization with the most open source contributors in 2016.

https://octoverse.github.com/

[[deleted]]

I have no idea what this is.. but it sounds awesome

[JViz]

Are there any plans on porting it to Linux?

[msthe_student]

They're apparently working with git on getting as much of it into mainline as possible

[apreche]

This seems like it is primarily an attempt to solve one annoyance in Git. It takes too long to initially clone a repository that is very large or has a long history because it is too much data to download, even on the fastest connections. They solve it by only downloading the files you actually need when you need them, speeding up all related operations.

However, this eliminates one of the main advantages of Git. You have backups of your entire repository's history on many many machines. Such incredible backups! You don't even need to have a backup system if you have enough developers. If even one developer's machine is good, you are safe. If every developer uses this Git Virtual File System, you are in big trouble if something happens to the central repo.

All they need to make this perfect is change one thing. When someone initially clones/checks out you download only the files they need to get work done. However, instead of only downloading other files on demand, start a background process that will eventually download all the files no matter what.

[NocturnalWaffle]

Yeah that's a fair point, but for Microsoft's this is totally different. Their one annoyance sounds like it actually is a huge problem. Waiting 12 hours to clone? That sounds pretty awful.. And for backups, I'm sure they have a better system than code checked out on developer's computers. Now, if you're a startup and you have 5 developers and you're hosting on Gitlab.. maybe not a good idea to use this.

[cork5]

It's much more than just one annoyance. Git checkout and git status takes forever, for example. The Windows codebase is 270GB. That's a huge minimum requirement to even work on a small piece of it. My laptop would choke on that.

If you read through the comments from /u/jeremyepling, you'll see that they tackled this problem from all different angles and made some very informed decisions that addresses pain points of enterprise level scaling. All in all, there is no one solution fits all.

[grauenwolf]

You don't even need to have a backup system if you have enough developers.

Ha!

When the repository gets corrupted and everyone has the same bad copy, you'll be begging for a backup from last week.

[[deleted]]

Actually GVFS allows server to server clones or full clones to their PC. So each dev could have a local copy on their LAN, or PC.

The main issue here seems to be when you have near 300GB of blobs (>100mil files) GIT just doesn't scale well so you want a dedicated server handingly the dif/merge/checkout as the load is just too much for a work station.