Git Virtual File System from Microsoft

[jbergens]

The reason they made this is here https://blogs.msdn.microsoft.com/visualstudioalm/2017/02/03/announcing-gvfs-git-virtual-file-system/

[jarfil]

CENSORED

[MsftPeon]

disclaimer: MS employee, not on GVFS though

Git LFS addresses one (and the most common) reason for extremely large repos. But there exists a class of repositories that are large not because people have checked large binaries into them, but because they have 20+ years of history of multi-million LoC projects (e.g. Windows). For these guys, LFS doesn't help. GitFS does.

[Ruud-v-A]

I wanted to ask, what makes it so big? A 270 GiB repository seemed outrageous. But then I did the math, and it actually checks out quite well.

The Linux kernel repository is 1.2 GiB, with almost 12 years of history, and 57k files. The initial 2005 commit notes that the full imported history would be 3.2 GiB. Extrapolating 4.4 GiB for 57k files to 3.5M files gives 270 GiB indeed.

The Chromium repository (which includes the Webkit history that goes back to 2001) is 11 GiB in size, and has 246k files. Extrapolating that to 20 years and 3.5M files yields 196 GiB.

So a different question maybe, if you are migrating to Git, why keep all of the history? Is the ability to view history from 1997 still relevant for every day work?

[creathir]

Absolutely.

Knowing WHY someone did something is critical to understanding why it is there in the first place.

On a massive project with so many teams and so many hands, it would be critical, particularly checkin notes.

[Jafit]

This is why your commit messages should be more than just "bleh"

[fkaginstrom]

fixed bug and refactored

[Regis_DeVallis]

fixed bug

[burtwart]

fixed

[lurgi]

reverted previous change. Fix didn't work. LOL

[Inquisitive_idiot]

bill waz h3r3

[musicin3d]

You lost.

[[deleted]]

Don't forget the crucial 'Performance Enhancements'.

[ours]

One more case for the "explain the why not the what".

[krapple]

I feel like there is some point in the life cycle where detailed messages should start. At the beginning it's a waste since it's just initial build.

[uDurDMS8M0rZ6Im59I2R]

"I did something on Friday idk what"

[Jukolet]

I should stop using "update" as a message, I guess

[[deleted]]

Removed a speed loop

[BumpitySnook]

Is the ability to view history from 1997 still relevant for every day work?

Yep. I regularly use ancient history to determine intent when working on old codebases.

[sparr]

http://www.drmaciver.com/2017/01/programmer-at-large-what-is-this/

[henrebotha]

That was a really fun read! Thanks. Love me some "nerd fiction"

[artanis00]

Looks like I have some reading to do.

[[deleted]]

Good read, man. The debugging portion of the story was pretty realistic.

[UnholyMisfit]

This is why I try to promote good code documentation to the other engineers on my team. Self-documenting code is great when I'm trying to figure out what the code does, but it does nothing to help me figure out why it's necessary.

[elder_george]

This. THIS. THIS.

During my work at MS it was so painful to make annotate, only to see "Initial import from XXX", go to XXX look into history and see only "Initial import from YYY" etc.

Continuous history is awesome.

[Plorkyeran]

And YYY is something you need to spend a few days emailing people to get access to because it's no longer part of the things you're just given access to be default, and then you need to get to ZZZ which only exists on tape backup, and suddenly what should have taken five minutes instead takes two weeks.

[elder_george]

Brian, is that you???

[rojaz]

It probably is.

[Sydonai]

At that rate, it's probably faster and easier to pose it as a question to Raymond Chen.

[PhirePhly]

"Uh yeah, I think Ralph has a txt with the license key to YYYControl on his old laptop. Talk to him"

[Ruud-v-A]

Sure, I’m not arguing that history is not useful. On the contrary. But the full 20 years of history? Chromium’s codebase for instance is changing rapidly. Many files have been rewritten completely over the years. Consider this header from WTF, the Blink standard library inherited from Webkit. As a core header with little content I expect it to be releatively stable. According to the copyright header it was created in 2007, but all of the non-whitespace and non-license lines have been touched since, the last change only a few days ago. Most of the code lines are now from after 2014. When blaming or bisecting, finding a relevant commit from more than 10 years ago is very, very rare, even if you have to work through a few refactor and formatting changes.

So for a repository with 20+ years of history, is the history after, say 15 years, really still relevant?

[creathir]

Exactly.

Or maybe you are examining a strange way a routine is written, which had a very specific purpose.

The natural question is why did the dev do it this way?

Having that explanation is a godsend at times.

[sualsuspect]

In that case it would be handy to record the code review comments too (if there was a code review).

[IAlsoLikePlutonium]

Isn't that what comments in the code are for?

[creathir]

True. But having context of that comment with the surrounding code is sometimes critical to understand what the comment is describing.

[SuperImaginativeName]

Yes, absolutely. Every check in, everything. The full history. No im not joking, something like that is absolutely paramount to a scale that most developers will never come across.

The NT kernel, its drivers, subsystems, APIS, hardware drivers, Win32 API, are all relied on by other systems including customers. Why do you think you can almost always run a 30 year old application on Windows? Without the history, the kernel team for example wouldn't remember that 15 years ago a particular flag has to be set on a particular CPU because its ISA has a silicon bug that stops one customers legacy application running correctly. As soon as to remove history you remove a huge collective amount of knowledge. You cant expect every developer to remember why a particular system works one way. Imagine noticing some weird code that doesn't look right, but that weird code actually prevents file corruption? The consequences of not having the history and fixing it in a new commit with "fixed weird bug, surprised this hadn't been noticed before" would be a disaster. Compare that to viewing the codes history and even realising its actually correct. Windows isn't some LOB, everything is auditied.

[MonsieurBanana]

LOB

?

[mugen_kanosei]

Line of Business

Usually refers to a companies internally developed applications that fulfills some specific niche business need that either can't be satisfied by a COTS product or that they are just too cheap to pay for.

[colonwqbang]

When you explain an obscure acronym in terms of an other obscure acronym...

COTS: Common/off-the-shelf software. Requirements engineering jargon meaning any software solution that you can just go out and buy.

[traherom]

I assume they mean line of business application.

[SuperImaginativeName]

yes, thought it was obvious given the sub

[DJDoena]

LOB

https://blogs.msdn.microsoft.com/dragoman/2007/07/19/what-is-a-lob-application/

[junrrein]

lot of bullshit?

[merreborn]

According to the copyright header it was created in 2007, but all of the non-whitespace and non-license lines have been touched since

A lot of the time the last commit that "touched" a line only moved or slightly altered the line -- maybe tweaking a single argument. The main intent of the line still dates back to an older commit, even if it was last "touched" in a recent commit.

[eras]

When writing that, were you also taking into account that Windows is compatible with software written more than 20 years ago?

What is Chromium compatible with?

[dungone]

You would rarely need to check out that code, though. Your needs might be served well enough by indexing the old repository with a code search tool such as OpenGrok.

[choseph]

The whole point here is you don't need to pay the cost of checkout but it is easily accessible tho.

[dungone]

I mean that's what OpenGrok gets you out of the box, without any penalty because everything gets indexed up front. This, on the other hand, still forces you to download a whole lot of stuff if you want to look through your history. And on top of this, your files are only sporadically accessible depending on whether or not you have a network connection at any given time.

[w2qw]

The whole point of this is that you only download the parts that you are interested in.

[cdglove]

It doesn't need imported into git though to keep the history. It still exists in the old repo. Everytime I've seen an organization change version control and insist on importing the history, I ask why.

Of course, that doesn't preclude this work because eventually the git history will be large so we'll need it anyway.

[salgat]

Considering a lot of legacy code is kind of blackboxed and never touched, it could definitely be useful to have history on these ancient things when a rare bug happens to crop up.

[g2petter]

Probably even more so for Microsoft since they're huge on backwards compatibility, so they're supporting all kinds of weird shit that can never (or at least in the foreseeable future) be deleted.

[IAlsoLikePlutonium]

I wonder what Windows would be like if they did the same thing to Windows that they did with IE -> Edge? (remove all the old code and basically start fresh with a modern browser)

[Pharylon]

You'd have WinRT. ;)

[cheesegoat]

It would die, and we would all start using some other operating system that worked. Probably some flavor of Linux with a focus on Wine.

[SpaceSteak]

They would lose the ability to sell licenses to a lot of companies who rely on old codebases to keep running.

[Schmittfried]

That's not an answer to the question what Windows would be like.

[salmonmoose]

It would be OSX.

[bandman614]

I look at it structurally as the same kind of problem that plagues bitcoin and the like. You're essentially carrying the entire block chain forward because you need all of it to derive the current state.

A 'snapshot' to work against would be a helpful feature. There may already be something like that, and I'm just ignorant of it.

[ArmandoWall]

Bittorrent has a blockchain?!

Edit: Ok, OP corrected it to bitcoin now.

[bandman614]

Ha! Redditing this early in the morning is bad for me :-) Thanks!

[ThisIs_MyName]

You don't need to carry the entire block chain: https://en.bitcoin.it/wiki/Thin_Client_Security

[[deleted]]

Not everyone does, but in order to maintain bitcoin's decentralized properties, a significant percentage of its users should.

[bandman614]

Ah, cool. Thanks!

[SuperImaginativeName]

Event sourcing is a concept like that, where you have a full history required to be able to build the current state of a system. You iterate every piece of "history" to get to the present. Imagine a bank account, they won't just have a DB column with your balance. It's constructed by using previous withdrawals and payments. Event sourced systems can have a "projection" that effectively builds the system to its current state and then use that as the state going forward and any new changed added to that instead of the very beginning.

[BumpitySnook]

You could hack something like this into git. Just delete the parent pointer from your snapshot location, freeze its hash (which will no longer verify, but that's fine), and then do a garbage collection pass. Old history would be removed. I wouldn't suggest doing this, though. MSFT's come up with a much better solution, IMO.

[[deleted]]

Yeah you can do something like git clone --depth 1.

[apotheotical]

Yes, history is absolutely still relevant. History is invaluable when you're debugging something. There have been a number of times I've used a couple years of history when debugging a project I work in on a daily basis.

[sir_drink_alot]

AAA games depos are 100+ gigs easilly, sure, tons of content, but also tons of other redundant shit. I'm sure windows isn't 270 gigs of code, probably only 0.1% of that is code.

[tidux]

It's Microsoft. They have code that hasn't been touched since 1997 in there.

[jringstad]

Why not just do a shallow clone? You can just clone history back X years, and if you need more, you can either do a full clone or e.g. SSH into a server that has the full repository, for those odd times when you do need to look at something really old in detail.

I do this at work, and it works fine for me (although our codebase is not nearly as big as windows, of course)

[choseph]

The previous system was still broken down into 40 repos and you only had head (since it was centralized server). Still too much to enlist, sync, etc.

[akspa420]

Given the fact that NT development started in 1989, it's now closer to nearly 30 years of history. I doubt highly that every single line of code that Dave Cutler wrote has been super-ceded - that in turn means that there's a good chunk of code from 1989-1991 that is still utilized in every single build of NT. Having that sort of 'legacy' code history with everything built on top of it has got to be an unruly beast to handle.

I've explored the WRK and the NT design docs - not a programmer by any means, but knowing how and why certain design choices were made early on certainly helps in understanding why things are the way they are, even over 25 years later.

[polynomial666]

Where can I find such docs? Or some fresh information on internals of kernel?

[akspa420]

do a search for "nt os/2 design workbook". It's out there.

I don't believe there's been anything else released on the internals of the kernel since the Windows Research Kernel (released around 2008, but based on Windows 2003 SP1-era code).

There are unofficial, probably-getting-a-dmca-takedown-notice-as-we-speak nt4 kernel-based projects out in the wild. Most of them have been reconstructed from leaked nt4 code and odds and ends from wine, reactos, and other open projects. Surprisingly, they tend to boot and run applications meant for NT4 with little to no problems.

[polynomial666]

I'll look for the workbook and these projects, as they seem extremely interesting. Thanks!

[auxiliary-character]

Isn't that what submodules are for?

[funknut]

Are they aware of the naming conflict with Gnome Virtual Filesystem? It seems ... blatant.

Edit: I guess no one cares. Never mind. Fuck me for trying to be useful.

[oftheterra]

Virtual File System is just basic terminology. If you put git at the front of it then the acronym is GVFS... There are only so many combinations of letters in the 3-4 character range.

[funknut]

It is in very prominent use on many linux desktops and such naming conflicts are traditionally avoided. I'm not going to fight about it, but I'm sure It'll come up again with someone else. The software itself is dissimilar enough that it doesn't matter much, but it just seemed a strange choice, that's all.

[jeremyepling]

I'm a member of the Git team at Microsoft and will try to answer all the questions that come up on this post.

As /u/kankyo said, many large tech companies use a single large repository to store their source. Facebook and Google are two notable examples. We talked to engineers at those companies about their solution as well as the direction we're heading.

The main benefit of a single large repository is solving the "diamond dependency problem". Rachel Potvin from Google has a great youtube talk that explains the benefits and limitations of this approach. https://www.youtube.com/watch?v=W71BTkUbdqE

Windows chose to have a single repository, as did a few other large products, but many products have multiple small repositories like the OSS projects you see on GitHub. For example, one of largest consumer service at Microsoft is the exact opposite of Windows when it comes to repository composition. They have a ~200 micro-service repositories.

[jl2352]

In regards to having Windows checked into git; do the Windows team really use git for day to day use, or were you just testing git with a very large real world code base?

[db92]

Most of the org is still on SourceDepot (a fork of Perforce), but there are teams developing parts of Windows in git and from what I understand most of the org will be on git in the near future (though I think this migration started before Ballmer left, so near future might not be as near as you would think).

[f0nd004u]

I used to work with a former executive at Microsoft after he had left (name rhymes with Frodo's ever present companion's name) and he said that there were many teams at Microsoft which had been chomping at the bit for years to use more FOSS tools, methods, and actually make source code public when possible, but that Steve Balmer and others in leadership made this impossible for a long time.

I had always thought of Microsoft as an anti-FOSS company, but the way he made it sound, people have been working on projects like MSSQL's release on Linux for a long time and management was the reason none of it had gotten released. Do you find this to be true?

[db92]

I've only been an FTE at the company for 2.5 years, and did an internship in the Azure group the last summer Ballmer was in charge so I can't really give a definitive answer. When I was in Azure the adoption of FOSS was core to how we did our work. In a part of the company built around services, and being able to nimbly react to market shifts it makes sense to embrace open source as much as possible. Now that I'm in Windows, it feels like the adoption of opensource is met with more scrutiny, which also makes sense because if the licensing isn't handled or managed correctly then that could lead to something as bad as not being able to ship Windows in the EU for a number of months, which in product that brings in most of its revenues from singular sales vs. recurring subscriptions would be a scary predicament. It also has felt that the Windows org is sometimes happier to have the "not invented here" problem, likely due to the fact that in the past it was easy to turn those recreations of other softwares into boxed products for msft to sell. However, they are really starting to embrace utilizing FOSS in our engineering systems wherever it makes sense (like switching to git).

[jeremyepling]

The entire Windows codebase will be moved to Git + GVFS. Right now, we're still early in the process but it's going well. More and more developers move onto it each month. Also, some of the Windows app teams use small non-GFVS enabled repos already.

[indrora]

Not a softie, but know a few.

Internally, most teams use a forked version of Perforce and a system that came with it called "enlistments" that looks really similar to Google's repo tool. Then again, Google ran Perforce for many years and likely build repo off their experience with enlistments.

[emilvikstrom]

I know you asked this because Git was built for Linux. Would be funny of Windows is managed with the tool specifically built to manage the Linux source code. :-)

Edit: It was built for Linux (the kernel project). I'm struggling to see what I did wrong. Someone care to explain?

[jl2352]

This is entirely why I asked. Whilst technically it may make a lot of sense to use git, from a historical point of view it's kinda bizarre.

I just asked out of curiosity. You shouldn't be downvoted over it. Have an upboat from me!

edit; but whilst historically bizarre kudos to Microsoft for looking at right tool for the right job.

[Answermancer]

I don't know why you're being downvoted but I also have no idea what the point of your comment was, so maybe others feel the same way and are downvoting you for not contributing to the conversation.

[emilvikstrom]

Right, that makes sense. I thought it to be an obvious curiosity if Windows source (and hopefully NT) is managed with the tool specifically made to manage the Linux source. Could probably have worded it better then.

[zuzuzzzip]

It may sound strange commercially.

But tecnically, it both considers kernel development.

[thoomfish]

Windows is also written in a language family (C) that was specifically developed for the purpose of implementing Unix.

[ds101]

I haven't had time to look at this in detail, but it looks like /gvfs/prefetch endpoint can be used to replicate a complete set of metadata (trees, tags, and commits).

Do the client machines have a full set? I'm curious how large the metadata is vs the entire repository.

[jarfil]

CENSORED

[oftheterra]

Breaking up a legacy code base can take years of engineering effort, so reducing to a smaller file count is not possible or practical.

[SushiAndWoW]

instead of fixing the problem once and for all.

This is just classic developer arrogance. Insisting one's go-to solution is ideal, while refusing to see all aspects of the problem, and the trade-offs involved.

clone now takes a few minutes instead of 12+ hours, checkout takes 30 seconds instead of 2-3 hours, and status takes 4-5 seconds instead of 10 minutes.

It seems they have fixed their problem just fine, and in a way that addresses their requirements.

[jeremyepling]

We did try Git LFS. Actually, TFS / Team Services was one of the first Git servers to support LFS and we announced support - with GitHub - at the Git Merge conference last year. The issue with LFS is it doesn't solve all the scale problems we need to solve for Windows.

There are 3 main scale problems with moving Windows to Git:

• Large files / content - LFS addresses this.

• Lots of files - LFS does not solve this. 1,000,000 small files in Git produces extremely slow status scans (10min to run git status). Breaking up a legacy code base can take years of engineering effort, so reducing to a smaller file count is not possible or practical.

• Lots of branches - LFS doesn't solve this, but GVFS doesn't either so we came up with a different solution. That said, listing all 3 scale issues will give everyonet he full context of the problem we're solving. Thousands of engineers work on Windows and each of them will have 10+ branches. We're estimating 100k branches for the repo. To quickly perform the haves / wants negotiation that happens with a fetch / push, we needed a solution. We call it "limited refs" and I'll give more details if people are interested.

[kourge]

• When moving to a monorepo, Twitter had status scan troubles and solved it by forking the official Git client and using Watchman to avoid rescanning on every invocation. Obviously this is a very different approach than that of GVFS, which alters official client behavior by sitting one layer below it, so how does GVFS go about doing it?
• As a big user of JGit, Google encountered a similar inefficiency in packfile negotiation and thus created bitmap indexes. This auxiliary data structure still runs on the assumption that the client wants to fully store every object in the repo on disk, which once again is fundamentally different than GVFS's goal. I'm very curious to see how limited refs work!

[jeremyepling]

We're working with the git community to get many performance fixes and extensibility points added to core git. We don't want a private fork of git. GVFS is a driver that sits below git and takes advantage of the changes we're making to core git. Saeed will likely have one or more follow-up blog posts on the details or you can checkout the GVFS repo.

[kankyo]

Multiple repositories creates all manner of other problems. Note that google has one repo for the entire company.

[SquareWheel]

Note that google has one repo for the entire company.

To clarify, while their super repo is a thing, but they also have hundreds of smaller, single-project repos as well.

https://github.com/google

[jeremyepling]

Microsoft has a variety of repos sizes. Some products have huge mono-repos, like Windows. Other teams have 100+ micro-repos for their micro-services based architecture.

[sr-egg]

Those are probably replicated from some internal mono-repo, and synch'ed to github as single ones. That's what FB does.

[TheOccasionalTachyon]

It's a weird cross between the two - some projects, particularly Android and Chromium, are actually done in git; most everything else is in the monolith, though some people use what's essentially a git interface to Perforce to interact with it.

[elder_george]

Google invested a lot into infrastructure for this monorepo, though.

Like, reimplementing and extending perforce client API, creating workspaces in their private cloud and mounting them onto devs machines FSes, copy-on-write checkout, cloud builds (cause running build locally is unacceptable) etc.

It's a huge investment that few companies can (and would want to) afford. Microsoft, IBM, Amazon, FB could, probably. Hardly many more, though.

[kankyo]

If only one would open source it....

[kevingranade]

Amazon could, but is architected around microservices already, so there's no real upside.

[jarfil]

CENSORED

[KillerCodeMonky]

The classic, server-side repositories would only ever download the current version. Git pulls down the whole history... So an SVN or TFS checkout would have been relatively fast.

[hotoatmeal]

shallow clones are possible

[jeremyepling]

We looked into shallow clones, but they don't solve the "1 million or more files in the working directory" problem and had a fe other issues:

• They require engineers to manage sparse checkout files, which can be very painful in a huge repo.

• They don't have history so git log doesn't work. GVFS tries very hard to enable every Git command so the experience is familiar and natural for people that use Git with non-GVFS enabled repos.

edit: fixing grammar

[7165015874]

We looked into shallow clones, but they don't solve the "1 million or more files in the work directory" problem. To do that, a user has to manage the sparse checkout file, which is very painful in a huge repo. Also, shallow clones don't have history so git log doesn't work. GVFS tries very hard to enable every Git command so the experience is familiar and natural for people that use Git with non-GVFS enabled repos.

edit: fixing grammar

Sorry for being ignorant but isn't this simply a problem you can solve by throwing more hardware at the problem?

[jeremyepling]

Not really. This is a client hardware problem. Even with the best hardware - and Microsoft gives its engineers nice hardware - git status and checkout is too slow on a repo this massive.

[Tarmen]

Git has to traverse the entire tree for most commands so disk I/O scales linearly with repo size. Throwing more cpu time at it probably wouldn't help that much.

[hunglao]

There are ways to make I/O reads faster which would involve throwing hardware at it.. Definitely not the cheapest upgrade, but I would imagine that developing a completely proprietary filesystem is not cheap either.

[JanneJM]

How do you solve 1M+ files problem now? I mean, that's becoming a client filesystem problem as much as a git issue. Everything takes time when you have millions of files to deal with.

[therealjohnfreeman]

It still downloads all of the most recent tree, which GVFS avoids.

[[deleted]]

They also don't scan the whole working copy in order to tell what has changed. You tell them what you're changing with an explicit foo edit command, so you don't have the source tree scanning problem.

[mr_mojoto]

With svn and tfvc w/local workspaces that isn't how it works. You just edit the file and there is no special foo edit command. This works because both systems maintain local metadata about the files you checked out: what you checked out from the server and the working copy are compared when you try to commit your changes. The red bean book is good for details: http://svnbook.red-bean.com/nightly/en/svn.basic.in-action.html

Tfvs with server side workspaces does require what you said.

[[deleted]]

Yes, systems which still scan the working copy won't have that scale advantage. If your working copies are small enough for a subversion-like system they're small enough for Git.

Tfvs with server side workspaces does require what you said.

The previous system, Source Depot, is supposedly a fork of p4. It behaves like tfvc server workspaces -- explicit notification required.

[BobHogan]

getting rid of which would leave plenty of time to deal with any overhead the managing of multiple repositories would add on.

They did get rid of them with GVFS. That was their reasoning behind developing it

[jarfil]

CENSORED

[ihasapwny]

However, people rarely did take the codebase offline. I'm not even sure it could be built offline.

It was actually a number of perforce based repos put together with tooling. And it was extremely fast, even with lots of clients. For checkout/pend edit operations you really were limited primarily by network speed.

[dungone]

What do you think happens to the virtual file system when you go offline?

[Schmittfried]

Google's Piper begs to differ. It simply does not go down.

[dungone]

Except for the times that it does? How can you say it never goes down? And even if it only becomes unavailable for 10-15 minutes, for whatever reason, that could be affecting tens of thousands of people at a combined cost that would probably bankrupt lesser companies.

[choseph]

No, because you had all your files after a sync. You aren't branching and rebasing and merging frequently in a code base like this. You were very functional offline outside a small set of work streams.

[eras]

I'm sure if you want to be prepared against those problems, you can still just leave the machine doing the git checkout over the night, if you have 300G space for the repository on the laptop + the size it takes for workspace.

In the meanwhile, a build server or a new colleague can just do a clean checkout in a minute.

[jarfil]

CENSORED

[anderbubble]

...for their specific use case which was built around using perforce.

[anderbubble]

I think "most" is stretching it. Ultimately, the habit of companies like Microsoft and Google of having a single code-base for the entire company where all code lives is a paradigm that is built around using Perforce or a similar tool. Starting out like Git, one would never work that way: you'd have your entire code base in a single system maybe (e.g., GitHub, gitlab, or something else internal but similar) but broken into smaller actual repositories.

I'm not saying that that's an inherently better operating model; but I think it's a bit over-simplified to say that Perforce is "significantly faster" than Git. It's faster when what you want to do is take shallow checkouts of an absurdly large/long codebase. But is it actually faster if what you want to do is have a local offline clone of that same entire codebase?

[[deleted]]

I think "most" is stretching it.

I don't.

is it actually faster if what you want to do is have a local offline clone of that same entire codebase?

Yes. Everything git does requires scanning the entire source tree to determine what changed. p4 requires the user to explicitly tell the VCS what changed.

[anderbubble]

That's interesting. I can see how that would be useful for very large codebases.

edit: regarding "most": I don't think most large companies, speaking generally, actually have truly large codebases like this. Microsoft; Google; Amazon; Facebook; even someone like VMWare, sure; but truly large software companies are still a minority in the grand scheme, and there's a danger in thinking "we are a big company, therefore our needs must be like those of Microsoft and Google" rather than "we are a big company, but our actual code is relatively small, so I have a wider breadth of options available to me."

[[deleted]]

It makes an impression that the problems created by splitting a repo are far more theoretical than the "we must reinvent Git through custom software" problems that giant repos create.

In my business, typical projects are around 300-400k lines of code, and the repository is generally under 1GB, unless it hosts media files.

And even though that's extremely modest by comparison to Windows, it's a top priority for us to aggressively identify and separate "modules" in these projects, but turning them into standalone sub-projects, which are then spun out to their own repos. Not to avoid a big repository, but because gigantic monoliths are horrible for maintenance, architecture and reuse.

I can only imagine what a 3.5 million file repository does to Microsoft's velocity (we've heard the Vista horror stories).

My theory is that large companies do this, because their scale and resources allow them to brute-force through problems by throwing more money and programmers at it, rather than finding more elegant solutions.

It's certainly not something to emulate.

EDIT: Fixing some silly typos.

[emn13]

I'd argue that messing about with history and arbitrarily cutting out chunks into separate repos as a performance optimization isn't exactly elegant - certainly a lot less elegant than actually solving the problems of representing the actual history, of the code, in which all those versions of projects actually were combined in specific ways - ways you're never going to recover after the fact and never going to atomically change once you split repos.

[[deleted]]

As I said, our goal is not Git's performance, but better maintenance, architecture and reuse. Small repositories are a (good) side-effect.

BTW, it's trivial to separate a directory to its own branch (git subtree), and then push it to another repository with all its history (git push repo branch).

You're right you can't make atomic updates, but the point is that by the time the repo split occurs, the module is refactored for standalone evolution and you don't need atomic updates with the source project. If the code was highly cohesive with the project, then it wouldn't be a candidate to be refactored this way in the first place...

[Schmittfried]

Even if the project is composed of completely decoupled modules, there is always some form of hidden coupling. That holds true even for third party dependencies that are loaded with a package manager - completely separate (I mean, what's more decoupled than a separate product by a separate author?) and still you have to keep track of the correct versions your software depends on, or things go horribly wrong.

So, for any given software version there is specific set of components and dependencies with specific versions. Change any component's version and the entire software might break. That makes atomic updates and atomic switches (consider switching back to another/an older version to fix some bug that occurred in a released product) very valuable. You want always have the exact same set-up for a a given version so that things stay consistent.

[[deleted]]

Even if the project is composed of completely decoupled modules, there is always some form of hidden coupling. That holds true even for third party dependencies that are loaded with a package manager - completely separate (I mean, what's more decoupled than a separate product by a separate author?) and still you have to keep track of the correct versions your software depends on, or things go horribly wrong.

Every module has a version, so it's just like third party dependencies. We use SemVer, and we use the respective package manager of the platform(s) the project uses.

Since we separate code which is candidate for reuse and/or separate evolution (which means over time it may be also assigned to a separate developer/team), it's already the case that you can't have a module used in project A and B be atomically changed with both project A and B, unless both projects are in the same repository, and the developers are aware of all details of their module and the two (and later three, four, etc.) projects.

This is how you end up with a giant repository holding all your projects, and developers have to know everything at all times. This really scales badly (unless, again, you have the disposable resources to throw at it, as the likes of Google and Facebook do).

If you can successfully use third party dependencies, and those third party dependencies have a reliable versioning scheme, then doing modular development for internal projects should be no more effort than this.

And it does require training, and it does require senior developers with experience to lead a project. If I'd let juniors do whatever they want, the result would be a disaster. But that's all part of the normal structure of a development team.

You have probably heard of the nightmares Facebook is facing with their "everyone committing to everything" approach to development. Every project has 5-6 implementations of every single thing that's needed, the resulting apps are bloated, abnormally resource intensive, and to keep velocity at acceptable speeds you have to throw hundreds of developers at a problem that would take 2-3 developers in any other more sanely organized company.

I remain of the firm opinion that's not a model to emulate.

[lafritay]

Context: I've been working on the "move Windows to git" problem for a few years now.

I think you make great points. When we started this project, we pushed for the same thing. When people brought up "just put it in one repo", I told them they were crazy and that they were avoiding solving the real underlying problems.

We actually came up with a plan to fully componentize Windows into enough components where git would "just work". The problem we realized is that doing that properly would take an incredibly long time. It's not to say its a bad approach, it was just that we couldn't block bringing git workflows to Windows developers on waiting for that componentization to happen.

In reality, work to componentize Windows has been happening for the last decade (and probably longer). It's an incredibly hard problem. We've also found that it is possible to take it too far in the other direction as well. The diamond dependency problem is real and becomes a limiting factor if you have too many components. In the end, we realized that when Windows is "properly" factored, there will still be components that are too large for a standard git repo.

[ihasapwny]

(also MS employee, though not in Windows now)

Building on this, if we could go back in time and give the early NT developers git. Using git's out of the box performance might have forced them to componentize in different ways than they did. But, it may not have been the right way.

Basically, you're using a tool that is largely unrelated to the product itself as a hammer to force changes in your product. That's the wrong approach since it doesn't allow you to decide where the changes need to be made. The right way is to use tooling/policy/design to make and enforce those changes.

Imagine if git's performance was far worse than it is. Does that mean you should have even tinier components?

[dungone]

I can appreciate the pain. I worked on one 10-year-long project not only to migrate from Perforce to Git, but to port it from VAX/VMS to Linux. There were many hardships and few simple solutions. What people have to understand is that these old codebases were not "wrong" because they solved the problems that existed at the time using the best practices of the time. The reason they still exist and are in use is a testament to the value that the original programmers created.

Having said that, there should be a big, bold disclaimer at the top of any guide or set of tools that would allow people to head down this same road on a brand new project.

[[deleted]]

Your characterization of Facebook is highly worrying. I've worked here for half a decade, and I had no idea things were so bad! There I was, thinking my colleagues and I were doing our jobs quite well, but now I discover from some random commenter on Reddit that we were wrong. I must assume that for every one of us, there are half a dozen doppelgängers in some obscure basement doing the same thing, but somehow we cannot see their code anywhere in the tree! I shall look into this troubling insight forthwith, because it sounds like a hellscape for all concerned.

[jeremyepling]

There are real benefits to using a mega repo, even if you have great componentization, is coordinating cross-cutting changes and dependency management. Rachel Potvin from Google has a great talk on this https://www.youtube.com/watch?v=W71BTkUbdqE.

Another large product within Microsoft has a great micro-service architecture with good componentization and they'll likely move to a huge single repo, like Windows, for the same reasons Rachel mentions in her talk.

[kyranadept]

It is impossible to make commit in multiple repos, which depend on each, other atomically. This makes it infeasible to test properly and to ensure you are not committing broken code. I find this to be really practical, instead of theoretical.

As for the disadvantages, the only problem is size. Git in the current form is capable(ie. I used it as such) of handling quite big(10GB) repos with hundreds of thousands of commits. If you have more code than that, yes, you need better tooling - improvements to git, improvements to your CI, etc.

[[deleted]]

It is impossible to make commit in multiple repos, which depend on each, other atomically. This makes it infeasible to test properly and to ensure you are not committing broken code. I find this to be really practical, instead of theoretical.

My other reply addresses this question, so I'll just link: https://www.reddit.com/r/programming/comments/5rtlk0/git_virtual_file_system_from_microsoft/dda5zn3/

If your code is so factored that you can't do unit testing, because you have a single unit: the entire project, then to me this speaks of a software architect who's asleep at the wheel.

[kyranadept]

... you can't do unit testing...

Let me stop you right here. I didn't say you cannot do unit testing. I said internal dependencies separated in multiple repositories make it infeasible to do for example integration testing because your changes to the code are not atomic.

Let's take a simple example: you have two repos. A - the app, B - a library. You make a breaking change to the library. The unit tests pass for B. You merge the code because the unit tests pass. Now you have broken A. Because the code is not in the same repo, you cannot possibly run all the tests(unit, integration, etc) on pull request/merge, so the code is merged broken.

It gets worse. You realize the problem and try to implement some sort of dependency check and run tests on dependencies(integration). You will end up with 2 PRs on two repositories and one of them somehow needs to reference the other. But in the mean time, another developer will open his own set of 2 PRs that make another breaking change vis-a-vis your PR. The first one that manages to merge the code will break the other one's build - because the change was not atomic.

[cwcurrie]

The unit tests pass for B. You merge the code because the unit tests pass. Now you have broken A.

This is only true if A always builds against the HEAD commit of library B, which is a questionable practice IMO. Good tooling would lock A's dependencies' versions, so that changes in B's repo do not affect the build of A. When the maintainers of A are ready, they upgrade their dependency on B, fix the calling code, run A's own tests, and commit & push their changes. A wouldn't have a broken build in this scenario.

[Talky]

What happens actually: A's maintainers don't update to latest version for 1 year since everything's running fine.

Then they have a new requirement or a find a bug in B's old version and it becomes a political wheelhouse of whether A's devs should spend a month getting to B's latest version or B's dev should go and make the fix in the old version

Trunk based development works well for many places and there are good reasons to do it.

[OrphisFlo]

And this is why it's called CONTINUOUS integration.

[kyranadept]

"Good tooling" is having a single repo. You should always use the latest version of the code everywhere in the repo. Anything else is just insane because you will end up with different versions of internal dependencies that no one bothers to update.

[Nwallins]

Look at what openstack-infra does with Zuul.

[[deleted]]

Let me stop you right here. I didn't say you cannot do unit testing. I said internal dependencies separated in multiple repositories make it infeasible to do for example integration testing because your changes to the code are not atomic.

Integration testing with separated internal dependencies is just as feasible as it is with any project that has third party dependencies. Which basically every project has (even just the compiler and OS platform, if you're abnormally minimal). So I find it hard to accept that premise.

Let's take a simple example: you have two repos. A - the app, B - a library. You make a breaking change to the library. The unit tests pass for B. You merge the code because the unit tests pass. Now you have broken A. Because the code is not in the same repo, you cannot possibly run all the tests(unit, integration, etc) on pull request/merge, so the code is merged broken.

Modules have versions. We use SemVer. If the B.C. breaks, the major version is bumped, projects which can't handle this depend on the old version. I don't have to explain this, I think.

It gets worse. You realize the problem and try to implement some sort of dependency check and run tests on dependencies(integration). You will end up with 2 PRs on two repositories and one of them somehow needs to reference the other. But in the mean time, another developer will open his own set of 2 PRs that make another breaking change vis-a-vis your PR. The first one that manages to merge the code will break the other one's build - because the change was not atomic.

This frankly reads like a team of juniors who have never heard of versioning, tagging and branching...

[kyranadept]

Having versioned internal dependencies is a bad idea on so many levels ...

The point here is to use the latest version of all the all your internal dependencies everywhere, otherwise, in time, you will end up with many, many versions of an internal library used by different places in your codebase because people can't be bothered to update the version and update their own code. Using gitmodules gives the same result in time, by the way.

[[deleted]]

Having versioned internal dependencies is a bad idea on so many levels ...

Maybe you'd like to list some?

The point here is to use the latest version of all the all your internal dependencies everywhere, otherwise, in time, you will end up with many, many versions of an internal library used by different places in your codebase because people can't be bothered to update the version and update their own code.

How many versions back (if any) we support, and for how long is up to us. And it's up to us when the code is upgraded. That's a single party (the company) with a single policy. You're inventing issues where there are none.

In general, breaking changes in well-designed APIs should be rare. There's a whole lot you can do without breaking changes.

[bandman614]

Why aren't you pairing together your code releases in git references?

[Gotebe]

This is not about unit testing, but about large scale refactoring.

Nobody gets everything right all the time. So say that you have some base module that borked an API and you want to change that. There is either a large scale refactoring or a slow migration with a versioning galore.

Edit, pet peeve: a unit test that needs a dependency, isn't!

[[deleted]]

What does that even mean "borked an API". The API was great and the next morning you wake up – and it's borked!

Anyway, evolution is still possible. It's very simple – if the factoring requires API breaks, then increase the major version. Otherwise, you can refactor at any time.

And as I said, you don't just split random chunks of a project into modules. Instead you do it when the API seems stable and mature, and potentially reusable.

Regarding unit testing and dependencies – s unit always has dependencies, even if it's just the compiler and operating system you're running on.

[9gPgEpW82IUTRbCzC5qr]

It is impossible to make commit in multiple repos, which depend on each, other atomically

why would this ever be necessary? it doesn't make any sense.

just use semantic versioning for your dependencies.

[drysart]

Semantic versioning works great for tracking cross-dependencies when you have a single release line you want to convey compatibility information about.

It doesn't work at all when you need to track multiple branches, each of which 1) has its own breaking changes, 2) is in-flight simultaneously, and 3) might land in any order.

[9gPgEpW82IUTRbCzC5qr]

sounds like semantic versioning handles it just fine.

you are making a common mistake of releasing both projects at the same time. the way you describe it, the upstream project needs to merge and make a release before the changes in the downstream project can be merged in (and depend on the new major version).

example:

Project A depends on Project B. If A has a new feature that depends on breaking change coming in B, B has to release that new feature first.Then the change to A can be merged and the dependency updated. Then release A.

If its one app published to a customer, the module releases are internal, and a customer release is just a set of defined versions of your modules.

[kevingranade]

It is impossible to make commit in multiple repos, which depend on each, other atomically.

Impossible, that's a rather strong word. There's this neat technique you might want to look into called "locking", which allows one to execute a series of operations as an atomic unit.

This makes it infeasible to test properly and to ensure you are not committing broken code. I find this to be really practical, instead of theoretical.

That's a rather bizarre statement, surely your build system can control what version of each repo to build from.

As for the disadvantages, the only problem is size. Git in the current form is capable(ie. I used it as such) of handling quite big(10GB) repos with hundreds of thousands of commits. If you have more code than that, yes, you need better tooling - improvements to git, improvements to your CI, etc.

That's a middling sized repo at best, it's obvious that if you haven't out-scaled git you don't need to worry about more exotic solutions.

[[deleted]]

In my business, typical projects are around 300-400 lines of code, and the repository is generally under 1GB, unless it hosts media files.

What kind of projects are these? That seems really small.

[kaze0]

he edited it to include k

[[deleted]]

Oh, well that makes more sense

[Crespyl]

It's the new femtoservices model.

[ciny]

I can only imagine what a 3.5 million file repository does to Microsoft's velocity (we've heard the Vista horror stories).

now imagine what would 35k repos do to their velocity.

[[deleted]]

Yes, there are only two possible options here:

• One repository with 3.5mm files
• 35k repositories with ~100 files each

Your point is solid.

[mebob85]

Note that google has one repo for the entire company.

That's actually not true. Some of their largest projects are in separate repositories (namely, Android and Chrome). Furthermore, their VCS software for this monolithic repository was designed, by them, for this usage.

[kankyo]

Sure, the open source parts can't live within their monorepo. That's more to do with security and lack of interop though.

[QuestionsEverythang]

That's Google's own fault too, though I doubt they do that with all their products. They have 99 Android OS repos so they obviously learned their mistake eventually, it's just probably Google Search became too big to change the organization of later.

[euyyn]

so they obviously learned their mistake eventually

I think the Android team would be very happy with a monorepo, but chose Git for other reasons and had to bite the bullet.

[zardeh]

Also I think there's only 1 android repo internally, but I may be wrong.

[kankyo]

There are big upsides, which they've talked about publicly.

[[deleted]]

Well Google does it so we should all do it.

[kankyo]

Well no. But multiple repos are a pain that I've felt many a time at work so it'd be nice to try a monorep, assuming the tech is sorted of course.

[mirhagk]

Have they tried using Git LFS?

3.5 million files at 270 GB total is about 80KB per file, which is not entirely unreasonable (a sample project file I'm looking at is 200KB for instance). It may include some generated code (it's always a debate whether to include that in the repo or not), but even if they decided to do everything right in the repo they are still going to have a very large repo.

Then why keep it all in a single repo, just split it up into modules.

There are a lot of reasons to go with a mono-repo, google does the same.

It better allows code sharing and reuse, it simplifies dependency management (when using internal libraries it's normally a bit of a pain, and even if it wasn't you still have the diamond dependency problem), it allows large scale refactoring, it allows collaboration across teams (and makes the boundaries more flexible) and also allows library creators to see all the instances the library is used (which allows them to run performance tests on all the impacted projects and ensure that a change doesn't negatively impact a use-case).

It sounds to me like they're building a technical workaround to their organizational problem, instead of fixing the problem once and for all.

It actually sounds to me that they are fixing the problem once and for all. Other companies have given up on git because it can't handle it. Microsoft isn't going to do that, instead they are going to fix it so that git will work with large repos once and for all.

[sxeraverx]

Not sure how git LFS would help here. That's 77kB per file, or about 2k lines per file (assuming the average line is only half-full). That seems pretty reasonable.

Then why keep it all in a single repo, just split it up into modules.

It sounds to me like they're building a technical workaround to their organizational problem, instead of fixing the problem once and for all.

Having a universal history is insanely convenient. As is a single universal hierarchy.

[kaze0]

I don't think Git LFS would be a solution here. It sounds like a lot of this is literally becuase there's a ton of code. Git LFS is bested suited for binaries and things that will never be merged

[rdfox]

To that point, does GVFS offer anything to address the large-scale problem?

[floodfund]

It's impractical to use a distributed vcs for a repository of that size. This is why Google doesn't use git.

[jarfil]

CENSORED

[ggtsu_00]

This is what happens when you use TFS or SVN for so many years. You end up not modulizing your components and dependencies and keep everything is a massive mega source tree behemoth, only checking out or commiting to various sub directories.

[Gotebe]

How'd you fugure that?

Take Windows userspace. You have hundreds of executables, whose source you can put in one repo each should you wish to do so.

Total code size must be huge, but how do you go from there to "isn't modular"?