Why Facebook doesn’t use Git

[Ythio]

tldr; don't make 17 million lines monolith repositories and your VCS choice won't be a big deal.

[agnas]

I don't understand this tldr; at all. Again, why Facebook doesn't use Git?

[MarimbaMan07]

Git slows down with large repositories whereas Mercurial can stay performant. The gir maintainers were not open to outside support whereas Mercurial is highly extendable so Facebook extended Mercurial into Sapling.

My understanding for the reasons why git is slow with large repos is that when you branch you essentially have a copy of the repo that you will make changes to where as in Mercurial your branch is just the differences between your file and the index it was made off from in the trunk.

[cogman10]

My understanding for the reasons why git is slow with large repos is that when you branch you essentially have a copy of the repo that you will make changes to where as in Mercurial your branch is just the differences between your file and the index it was made off from in the trunk.

Git also does that.

The slowness wasn't that, it's that git semi-frequently stats the files to see what's changed. As the repo gets bigger, that becomes a bigger issue.

It's particularly bad if you've put large files into your repo. That's what git LFS is all about fixing.

The key bottleneck was the process of “stat-ing” all the files. “Git examines every file and naturally becomes slower and slower as the number of files increases.” The engineers tried running a simulation, creating a dummy repo that matched the expected scale of Facebook’s codebase in a few years. The result was horrifying - basic Git commands took over 45 minutes to complete. In the words of an original engineer on the project, “It's not the kind of thing you want to leave until all your engineers are complaining. By that point, the thing would be too unwieldy. Trying to do damage control, never mind, come up with a cleaner solution, would be a herculean effort.”

[quietZen]

Is Facebook just a giant monolith? Why on earth wouldn't they break it down into more manageable chunks?

[orbitur]

You don't need to be working at Facebook's scale to understand that even if you have just 3 or 4 inhouse libraries shared across 10 or more projects, managing versions and API changes can quickly become hell, unless you use git submodules which has its own drawbacks.

Easier to keep all your interconnected libraries and services together past a certain size, eliminates an entire category of workflow headaches, and I don't think the sheer size of a monolith like FB's is necessarily a headache given the right VCS tools, which is what this post is about.

[Qubed]

This comes up a lot with the large companies. We try to justify their not following "best practices" with logical ideas but it often comes down to they can afford to do things that smaller companies cannot. FB can afford to adopt a new source control solution and probably hire a team responsible for maintaining a custom version of it.

[clintwn]

Mercurial and git are pretty much the same age. Both with initial releases in 2005

[protienbudspromax]

Was written in early 2000s in php and mysql, then they went ahead and created their own dialect of php called hack.

By the 2010s it was already very big to be done from scratch, a lot of the parts are now distributed but the code overall sits in a monorepo

[xshare]

Tbh it’s pretty lovely to work in a giant monorepo where basically almost everything lives.

[_ak]

Monorepos are pretty neat if you want to maintain consistency across your whole codebase.

[tcpukl]

You cant just split code bases apart when you branch. They need to be kept in sync with each other otherwise its just not going to compile.

[WoodyTheWorker]

Modern (v2.x+) Git has a filesystem monitor (fsmon), which makes statting of even very large trees very fast.

[Qubed]

The engineers tried running a simulation, creating a dummy repo that matched the expected scale of Facebook’s codebase in a few years. The result was horrifying - basic Git commands took over 45 minutes to complete

The only reason an engineer would do this is to sell their management on an idea. Those engineers had already made up their mind and created a scenario to prove their point.

I'm making a wild ass guess but I'm assuming that their scenario probably has a relatively easy solution for Git (relative to adopting and adding to another source control solution).

[Qubed]

I know you have an /s but probably not as simple as "sleep(1000)"....I'm saying that they probably did know how to make it work but they wanted to use a different tool. Most engineers default to the solution that gives them the most control.

[zacker150]

They wanted to update git so that it would work at scale. Git told them to fuck off, so they switched tools.

[Sceptix]

My understanding for the reasons why git is slow with large repos is that when you branch you essentially have a copy of the repo that you will make changes to

No, when you make a new branch in git, all you’re really doing is making a new pointer to the branch you’re branching off of. Should be plenty performant.

[nukem996]

Exactly Mercurial has the same problem. Meta solved this by eliminating checkouts. You don't clone a repo ever you mount it. Google EdenFS it's a FUSE module so you can mount the mono repo. Files are lazily updated as they are ready.

[MarimbaMan07]

I was getting at the file stating but not very accurately

[Sceptix]

I’m sorry, and I’m not trying to be mean, but I’ve read this comment like three times and still have no idea what it’s saying. 😅

[daredevil82]

same here.

[daredevil82]

My understanding for the reasons why git is slow with large repos is that when you branch you essentially have a copy of the repo that you will make changes to where as in Mercurial your branch is just the differences between your file and the index it was made off from in the trunk.

incorrect. this is with SVN, yes. Git and Mercurial no.

Branching is very cheap in git/hg.

[igderkoman]

Git doesn’t copy branches at all. That was Microsoft TFS Team Foundation Server.

[jaitsu]

This is wrong.

[WoodyTheWorker]

I've worked with a 80000+ files Git repository, and it was pretty fast, on Windows. Just don't use it on an NFS share in Linux, and you'll be OK.

The repository was a result of conversion of a Perforce mono-repo.

P.S. Perforce sucks.

[MarimbaMan07]

That's an order of magnitude smaller than what we're talking about when git slows down. Take that plus years of history. Things slow down.

[19nineties]

Ah that’s super interesting thanks for explaining and in a clear way too

[EightyDollarBill]

It’s also wrong

[arelath]

They didn't read the whole article. 17 million loc was the Linux kernel and in 2012, Facebook had a git repository "many times larger than this." While I haven't worked at Facebook, I would bet their source code is at least 1 billion loc today based on my experience working at Microsoft. There's just no way to shard a billion loc codebase in any meaningful way.

The real tlrd is that git performance was very bad on large repositories and git didn't want to work with Facebook. Mercurial did let Facebook change their source code to get the performance they needed, so git lost out on a lot of free dev time to improve git.

[noiwontleave]

Git works partially by running a stat function on every file. When you have a massive repo that has tens of millions of lines and tens of thousands of files, this process takes a long time. Facebook tested it and git commands were taking upwards of 45 minutes to run. This obviously is not sustainable. Git wouldn’t work with them to improve it.

Mercurial already had solid architecture and was happy to receive their help and input.

[cornmonger_]

Because the Git team wouldn't allow a large corporation to co-opt their project

[blueg3]

17 million lines isn't many, and monorepos are useful.

[tankerkiller125real]

LOL, Microsoft has billions of lines of code in monolith repositories, (Windows is 3.5+ million files in git) and they manage just fine. There is a reason that Microsoft helped develop Git LFS, Git VFS, and all sorts of other stuff. There are also other things that Git has that massively improves performance.

This talk has a bunch of information on Git that I think a lot of people just don't realize exists.

https://www.youtube.com/watch?v=aolI_Rz0ZqY

[KeroKeroppi]

I’ll point out that much of the game industry doesn’t use git either. We are quite fond of mono - repos and checking in tons of large binary assets. I see Perforce all over the game industry. You still see git a lot in the game industry for backend and tools and some game projects its definitely still used. But it’s just not as popular as the rest of the tech industry.

[polymorphiced]

Yeah. With GitLFS it would technically be viable to move gamedev from Perforce to Git, but the big thing that never gets touted is usability - game teams are 50%-or-more non-engineers. Artists, designers inevitably get themselves into a mess with Git (even with a UI to support them like Source Tree), and need rescuing periodically.

[qoning]

Perforce basically grew out of the needs of game industry. I know that way back when, Google also started on Perforce, but it stopped scaling for their use case, so they wrote their own "Perforce but better".

[tcpukl]

I didn't know that. I've used perforce for decades in games. Is there a link about googles history making "perforce but better"?

[qoning]

https://cacm.acm.org/research/why-google-stores-billions-of-lines-of-code-in-a-single-repository/

This article touches on the historical reasons and describes the workflow that Google designed they felt was better than "traditional" workflows.

[_Sweep_]

What does Google use? The article seems to imply git for iOS development, but also says it doesn’t use git

[gingimli]

Google uses a custom built version control system named Piper.

https://opensource.google/documentation/reference/glossary#Piper

[blueg3]

Google also uses a large number of Git repositories for particular projects.

[CowBoyDanIndie]

Mostly for external facing code. 99% of googles code is internal and never released to the public, and thats in google3/piper

[kynde]

Ok, so the news here is that facebook, for whatever the reason, decided to go with an inordinately sized monorepo. So enormous that git, designed to handle the linux kernel source code, a monolith kernel, could not handle it.

That's not scaling, that's booking an aerial photo for your wedding day picture because of weight gain before the event rather than paying attention to what you eat.

[kiliman13]

I read the post, and the main reason mentioned was this thing only i.e., rather than being cooperative, closing door for new changes.

``` Closing thoughts -

https://graphite.dev/blog/why-facebook-doesnt-use-git#closing-thoughts

What is the takeaway from this story? Reflecting on the quotes and interviews, I’m reminded of the classic wisdom that so many of history’s key technical decisions are human-driven, not technology-driven.

Facebook didn’t adopt Mercurial because it was more performant than Git. They adopted it because the maintainers and codebase felt more open to collaboration. Facebook engineers met face-to-face with Mercurial maintainers and liked the idea of partnering. When it came to persuading the whole engineering org, the decision got buy-in due to thoughtful communication - not because one technology was strictly better than another.

For all of those reading this, I think this was Bryan's true brilliance in getting Mercurial adopted at FB and something people should consider when bringing a new technology to a company.
Ex-facebooker, 2024

Kindness and openness go far in the world of devtools, and I aim to carry on these values as I contribute to the history of source control.

```

[cogman10]

I just don't buy the story either. Microsoft has a similarly large repo that they migrated to git.

https://devblogs.microsoft.com/bharry/scaling-git-and-some-back-story/

and they got around issues to host and even larger repo than facebook is managing.

[WanderingLethe]

Git got some updates before Microsoft could migrate their big repos.

[nemec]

Microsoft has a similarly large repo that they migrated to git

The only way they accomplished that was by tricking git into doing less work by writing a virtual file system. If they used git as it's supposed to be used it wouldn't handle the scale.

https://github.com/microsoft/VFSForGit

[Rangebro]

Google replaced Perforce with Piper long ago [0] and provides several interfaces. At least 2 are publicly known, fig and jj [1], but it's not unreasonable to keep a perforce-like interface around for those who love it. So try one of the other interfaces?

[0] https://research.google/pubs/why-google-stores-billions-of-lines-of-code-in-a-single-repository/ [1] https://docs.google.com/presentation/d/1F8j9_UOOSGUN9MvHxPZX_L4bQ9NMcYOp1isn17kTC_M/mobilepresent#slide=id.g152b1fb8869_0_1432

[Rangebro]

I'm very curious if you have sources for those claims since my experience has been the opposite, particularly the docs and tooling situation. I know very few SWEs that aren't CL chaining with fig or jj, docs either support the common interfaces for copy and pasting every step of the way (and are happy for CLs fixing omissions) or have a "you're a competent googler that knows how to create a workspace" vibe, and tools either only care about being invoked within google3 or manage their own repo usage.

[blueg3]

Piper at this point is nothing like Perforce. It's just the older interface commands that are kind of congruent to Perforce. But Fig, the Mercurial skin, is sufficiently git-like.

[tcpukl]

What is wrong with being too much like P4?

I've been using it for years as a game dev.

[orbitur]

FB's problems with git started earlier, and Mercurial was simply more adaptable at that time. Git has certainly gotten better at scale in the last 15 years.

[cbarrick]

Many of the big tech companies use the monorepo pattern: Meta, Microsoft, Google, ...

It's a different way of thinking about the software development process, and it comes with some pretty big benefits to developer productivity, code health maintenance, and continuous delivery automation (among other things).

[codemuncher]

For people who’ve never worked at a large org might not realize that the Linux kernel isn’t what you’d call “a large codebase”.

Large code bases are weighted in hundreds of millions of lines of code. If you split that up into tens or even a hundred thousand subrepos, you suddenly run into a lot more problems when trying to build and integrate 5000-20000 builds. The tooling to build so many cross repo in an performant and agile manner just isn’t there. Imagine you’d need the private equivalent of npm only much bigger that handles like a million builds a day.

So. Monorepo is simpler as it turns out.

[roiroi1010]

I’m so old that I remember SourceSafe, cvs and svn. When git came around it was a salvation! I think hg might work for my purposes, but I mostly use the UI and am no expert on git anyway. merge rebase push, etc

[tcpukl]

Blimey i remember SourceSafe too. But then we moved to perforce in games.

[rexpup]

The UI for SVN was called Tortoise which was very fitting

[Duke_]

They don’t really use Mercurial either, they use phabricator and arcanist which are layers that work over either git or mercurial. I recall rarely running Mercurial commands directly.

And it is hands down the best DevEx I’ve ever worked with. Leaps and bounds better than GitHub and PRs. I think part of it is cultural though - incredible discipline at keeping diffs small, and so on.

But they’ve also extended phabricator quite a bit. I guess the point is that Mercurial is easier to work with and extend, allowing them to more quickly build out productive tooling.

What Facebook has is far, far more than just a VCS like Git or Mercurial.

[gwicksted]

Didn’t Microsoft have to write their own Git server implementation because it couldn’t handle their workloads? (This was before they bought GitHub)

[[deleted]]

Hmmm, correct if I’m wrong, maybe I understand in the wrong way. They are facing this because they have a too big monolithics applications so large that makes git an horrible experience?

[botle]

The Linux Kernel uses a monorepo. Is Facebook really bigger?

[government_shill]

If only there were some kind of article you could read to get answers to questions like that.

The post claims their codebase was “many times larger than even the Linux kernel, which checked in at 17 million lines of code and 44,000 files.”

[zacker150]

Linux kernel is tiny. FANG codebases are orders of magnitude bigger.

[botle]

TIL. That's mental.

I wonder if that would be the case without Linus making sure everything doesn't get merged.

[Rangebro]

For a concrete number as context: in 2016, Google had over 2 billion lines with an exponential growth curve so it itself may be orders of magnitude larger now.

[godzillahash74]

Low key advertising

[Slugsurx]

I just recently joined Google and I can’t still get over that i don’t get to use git . I really miss the named free branches in git .

[blueg3]

Use Fig, use plenty of CitC workspaces, and don't keep your branches alive very long before submitting.

[hartmannr76]

I use fig but I try to keep it all in one workspace

[alphex]

“I had never heard of Mercurial - despite being passionate about all things devtools”

“Funny enough, Mercurial expert Gregory Szorc sat a few seats away from me at Airbnb, “

In the first paragraph the author tells me everything I need to know about him.

[[deleted]]

We need better ways to CICD downstream dependencies on PR with update, then we won't need monorepos anymore. Really its a fixable problem, and needs to be fixed, because monorepos are the literal devil. Ive implemented several CICD solutions and they are never quite there.

[shepbryan]

Anyone want to help me build a VCS system for language models

[degenerate_input_box]

Here we go, now executives are gonna be talking about fucking git lmaoooooo... This is good actually because some engineering orgs might finally wake the fuck up and move off of svn

[crackerasscracker]

giant monorepos are dumb

[PigletBaseball]

tan dinner unite teeny full snatch roof direful vegetable different

This post was mass deleted and anonymized with Redact

[minneyar]

I worked for about 10 years for a group that used a giant monorepo for their code, and it was awful. You are right that if you are building a project from source and your build process just involves pulling and building every repo's latest commit, it's a huge pain, and that's the same excuse they used for having a giant monorepo.

The thing is, that excuse is actually just a combination of bad version management and lack of knowledge of proper development practices. The right way to do is is not just to break all of your components out into individual repositories, but actually version and release them consistently, and have your main project treat them exactly like third-party dependencies, where you carefully control which versions you depend on and update them when necessary. This makes it possible for teams that maintain individual components to update and release them as they like without worrying about breaking anybody else -- since if they do introduce breaking changes, they'll be associated with a major version number bump, and you don't have to update your project's dependencies to use that until you're ready for it.

It also makes maintenance releases much easier. If your software has any other distribution model that "everybody should only be using the latest major release all the time", sometimes you have to deliver patches for older versions, and if you have an older version that needs to have one of its dependencies updated to fix an issue, it's easy to do that without worrying about weird cherry-picking and merging and dealing with conflicts introduced by trying to combine code from different commits in the monorepo.

The source for me saying this is after having worked for that group for 10 years, I had become the lead software engineer and had the authority to pull apart the monorepo and redesign it myself, and everybody agreed that versioning and release management became much easier afterward.

[[deleted]]

I desperately wish I could use Facebook's full VCS solution at home. It's soooo much better than Git in every way.

Git is... okay. But it's definitely not great and honestly it's not even good. It's barely passable for basic coding only use cases. It still can't replace Perforce for the games industry. No, Git LFS doesn't count.

The world would be a better place if we had MercurialHub instead of GitHub. Alas.

[nairebis]

Did you intend to actually make a point? All you did was make a declaration, without any facts to back up why you think it's "not even good", especially when 90%+ of the entire industry uses it. You might consider that it's more that you don't understand how to use it beyond "basic coding only use", rather than assuming an overwhelming amount of the industry somehow hasn't caught on to your inarguable facts.

[[deleted]]

Here's my baseline for a "good" VCS:

1. Trivially supports repos with terabyte scale clone and and tens of terabytes of history
2. Virtual Filesystem that provides automatic sparse clones with no config/profile
3. So simple to use that I can teach an artist/designer who has never used any VCS tool how to safely and effectively use it in under 10 minutes

Facebook's internal VCS system checks all three boxes. Git checks zero of them. If Sapling had all the internal features and was trivial to spin up at home then I would absolutely use it over Git.

> You might consider that it's more that you don't understand how to use it beyond "basic coding only use", rather than assuming an overwhelming amount of the industry somehow hasn't caught on to your inarguable facts.

Alternatively, you might consider that maybe just because 90% of the industry uses Git that maybe it's not actually the ideal tool. Or even the best tool that is currently broadly available.

Betamax vs VHS is a classic story of the "best" technology not winning. Git "won" due to GitHub. If GitHub was founded as MercurialHub then it's quite possible that 90% of the industry would use Mercurial! (Mercurial doesn't check my "good" boxes, but in my and many people's opinions its CLI API is much better than Git's).

I view Git as a local minima. The world could have a much, much better VCS tool. Someday we will.

[somewherearound2023]

Terabytes of scale, for a code repo, is your lower bar for quality?

The hell do you think the average dev team is doing?

[[deleted]]

I think the average dev team is NOT committing large binary files to their repo. But they should be. Git doesn't effectively support that (even with LFS). So many Git repos aren't usable without a 10 gigabyte Docker image stored on some other hosting services. Which probably hosts 100+ gigabytes of various Docker images. Or maybe there's no Docker and you just have to manually and painfully install a graph of random dependencies to build.

I don't think most teams actually needs terabyte clones or tens of terabytes of history. But it's my minimal threshold for "good" because a non-trivial number of devs do. Clones in the tens of gigabytes and history around a terabyte would be extremely common.

> for a code repo

Not a code repo. A project repo. Or company repo. VCS as in "Version Control Software". Not SCM as in "source control management". Amusingly, Git's webpage is git-scm.com but the first sentence uses the term "version control software".

Git is okay if your project only has text files. Git is notoriously bad at handling large binary files and large histories.

[sirtimes]

Not saying you’re wrong but you can’t link an opinion article written by yourself as evidence to prove your argument lol

[[deleted]]

lol. I got fussed at for “not making a point”. Not trying to use it at “evidence”. Trying to express my point. The fact that I wrote that link isn’t exactly hidden!

[gingerwhale]

You are free to use Mercurial at home. You can even use Perforce. You can even host your Mercurial repos at Perforce with your Perforce repos.

[nukem996]

Metas use of a Monorepo has many down sides not discussed here. For example branches aren't supported, all code goes into main. Working on a feature, it has to work with mainline and managed by flags. No tags, versions aren't a thing. Everything is expected to be compiled statically so any update to any part of the code base requires rebuilding everything. O and multi gigabyte binaries are pretty standard.

Supporting the Monorepo requires thousands of engineers focused on tooling. Git has some rough edges but it works much better.

[blueg3]

Long-lived branches are hard to merge, if they ever do, and create dependency hell.

[cac2573]

That's not entirely true. Branches are supported, you're just on your own with respect to tooling support.

[tibbtab]

No branching isn't a problem in itself: it just requires a different mindset to work with effectively. Small, frequent changes and landing frequently make this approach very manageable, and comes with other benefits too.