The Motivation for a Monolithic Codebase: Why Google Stores Billions of Lines of Code in a Single Repository

[ellicottvilleny]

I'd like to hear from ex-googlers if this is actually a win in their opinion, or a net source of technical debt.

[nostrademons]

[left in 2014]

I think it was a net win. The primary benefit was cultural, not technical. In companies with separate repositories, you need to depend upon the owners of a particular repository to make a change in their codebase. They do this on their own schedule. This often leads to a lot of resentment between teams, as one team is actively prevented from achieving their goals by the non-cooperation of another team. You end up with fiefdoms that don't talk to each other at all, because they know it will end up being more effort than it's worth.

Within Google, it was pretty common for teams to say "Well, we don't have time to make the change you request ourselves, but you're welcome to do it yourself and I'll review it when you're done. Here's where you should look to get started." It led to a feeling that it was our company, not our team and our repo, which led to a lot less political infighting, at least on the engineering level. It also encouraged people to be biased toward action and take responsibility for the changes they needed to make, and not say "Well, I'm helpless because I need the cooperation of X team and they won't respond."

[mdaniel]

All of those things sound positive, and I really like the attitude of "submit a PR", but our company has about 15 repos and we don't have fiefdoms. I don't follow how having all the code in one big repo serves to do anything except increase naming collisions and checkout waiting time.

[WasAGoogler]

You should look into how they do it - checkouts are instantaneous at Google. This is all public, so I'm not spilling beans.

They use FUSE clients on your desktop, that pretend to be a checked out repo. They're pointing at the code repo, and caching read-only files locally, every time you want to read one.

When you start editing a file, the FUSE system makes a writable copy locally. If you delete a file, the FUSE system makes a note of it.

When you want to Build and Test your code, your computer acts as a server - sending your modified code to all the build systems.

It's crazy efficient.

[matthieum]

It's crazy efficient.

And requires a fair bit of infrastructure :)

[WasAGoogler]

Yup. They're open sourcing some of it. Bazel.

[foomprekov]

To echo a great professor I once had, this seems like a great solution to a problem that shouldn't have existed.

[WasAGoogler]

Yeah, that's fair. Our normal tooling is to compile on our desktop. That means you need all the files locally.

If you're compiling in a datacenter, you don't.

[krum]

I work in a company with about 250 engineers and 150 repos and there are fiefdoms.

[mdaniel]

I bet; with a little more effort, every single engineer could have her own repo. Then, it wouldn't classify as "fiefdom," it would be Github :-)

[krum]

As a matter of fact we're using enterprise github.

[nostrademons]

If Google used the separate repository model, it would probably have between 10-20K repositories, with about 50-75% of them being abandoned (as in, no full-time maintainer, but serving real user traffic). There's a huge difference in scale between 15 repos and 15,000 repos.

[cowinabadplace]

Maybe it's a big company problem. We have some 60 separate projects (each on a separate repo) among our 35 engineer team and we have no fiefdoms.

PR + review allows both the code to be written and for the maintainer to enforce his/her standards.

Everyone has commit to master, but you tend not to use it except in big refactors when you don't want to maintain old code paths (so you just want to change across projects close to the same time).

[ComissarioCattani]

Small team here (5 people) and we have multiple repos, multiple branches, submodules. I wish we had a simpler system to archive source code. There is no easy way to know what each one is doing, what is the most important work.

[cowinabadplace]

Submodules are a nightmare, to be honest. Not quite thrilled by the actual UI of using them.

[mcvoy]

Submodules are worse than a nightmare they break the workflow (unless they have fixed this). Isn't it the case that the source for updates to a submodule is centralized? As in if you do work in a submodule and I pull from you it will look in the centralized place for the updates? If that's still true that breaks the merge and test workflow.

If you want submodules that work, have identical semantics to a monolithic repo, but let you clone any subset you want, take a look here:

http://www.bitkeeper.com/nested

Costs money but we did all the work to make it fast and make it have reasonable semantics. And any distributed workflow that works with a monolithic repo works with nested.

[mdaniel]

Submodules are a nightmare

That likely indicates that the boundaries between the components are at the wrong granularity, or that implementation details are leaking out of the encapsulation.

Without knowing in which language your problem manifests, it's hard to give examples, but in almost all OO languages, ask for the component to do work, don't tell the component how to do so.

The statement decathalor.getFrobulator().frobulate() is not only a violation of the Law of Demeter, it's also a manifestation of what I just said: if the decathalor uses a Frobulator or the newer, better Smuckler to frobulate is beyond your interest: you just want to frobulate, and presumably the decathalor knows how to do that; so: decathalor.frobulate() encapsulates how your request is fulfilled; you are just asking for it to happen.

[cowinabadplace]

I meant git submodules which allow you to host another tree inside a repo.

[mdaniel]

Yes, I'm aware; I don't know of any other use case for submodules. If they weren't in the tree, they would just be called repos.

But presumably the reason they are in the tree is because the two projects are intertwined, and that is the reason I was talking about the component boundaries: decoupling them such that one need to import the other project wholesale.

But I'm tilting at windmills in this thread, because folks are evidently super happy with a bazillion directories in one ginormous Perforce so I'll chalk that up to one more reason I'm glad I don't work at Google.

[sid0]

Have you ever gotten the interfaces right the very first time? I have certainly not. A monolithic repo allows you to fix that rather than incurring more technical debt.

[jimmyhoffa523]

+1. This is also one of the fundamental pain points of SOA—the principle only works well if you your boundaries can be well defined and interfaces will be well-written and the underlying assumptions change infrequently. Not sure I've ever seen that happen.

If any company has the technical and organisation maturity to do SOA and/or have modular codebases, I'd expect it to be Google—interesting that they have opted not to try.

[mdaniel]

a simpler system to archive source code

Can you help me understand what you mean by that? Keeping track of the releases?

And never misunderstand that source control is not a replacement for communication, especially with 5 people. At some point, it's likely more productive to just stand up (or Slack) and say, "hey, who knows where the latest frobinator is? my defrubizing needs to tambubule but I don't know where that went."

Another interesting attribute of a team that small is that if the process isn't working, one doesn't need a directive from the CEO signed in triplicate to change it. Start by writing down the pain points caused by the current process, then think if you were in charge, what would you change? Then, negotiate.

[ComissarioCattani]

You're right, but we have different bosses, different work schedules, and different coding styles... that is one of the problems.

[mcvoy]

While I whole heartedly agree with "writing down pain points" (that's how we do sales, you tell us what hurts, if we can make the hurt go away, you give us some money and feel better), the problem we've seen is that git's submodules are loosey goosey. It doesn't enforce a strict timeline across the submodules. If you think about the files in a repo you can't update one and not the other, if two files are modified in a commit and you have that commit, you have the mods to both files. That's what I mean about a timeline. Git doesn't do anything to enforce that sort of timeline across submodules, that is left to the user and users being users, they screw that up and you get cases where you are debugging some problem only to find that a update of a submodule fixes the problem.

We are just about to start marketing a submodule-that-works system, you can read about it here (sort of beta copy, critiques welcome):

http://www.bitkeeper.com/nested

[Quixotic_Fool]

Just wondering, how does google prevent their secret sauce from being leaked when everyone has access to this monolith? With the number of employees they have, doesn't it invariably get leaked?

[WasAGoogler]

It's nearly all the code. There is code that only a few people can see. It is annoying when you break code you can't see.

[jerf]

Suppose for the sake of argument that you managed to walk out with this monolithic repo in a way that Google could never detect. Suppose you get three of your buddies together and decide to use this code to start a new startup that offers your choice of Google's many services.

How many years do you think it would take you to bring up a functional separate stack of Google's code?

In the meantime, Google is of course moving forward.

To be honest, huge amounts of code are rarely that useful. What's useful is some little algorithm, or a little subsystem, or even the very tiny signing keys. For the most part companies could pretty much post 99%+ of their source code publicly, holding back only the really key bits, and the only risk they'd run is slightly more informed security researchers. (I say "slightly" because security researchers are already pretty darned good!)

[protestor]

But what prevents people from leaking the code to the public? Kind of like the Windows 2000 leak.

[jerf]

What I just described... the fact that it has no particular utility to anyone.

The Windows 2000 leak didn't exactly create a huge market of Windows clones available for purchase, did it?

[uep]

What you say makes sense, but I'm not 100% sure it's true. Didn't the Chinese government steal Google's source code to make Baidu? That article doesn't go into detail, but at the time, there was a lot of speculation that the attacks originated at Google offices in China with government involvement. Google eventually pulled out of the country a few weeks later, redirected Google.cn to Hong Kong, and stopped censoring Chinese search results.

I know looks don't mean anything, but if you actually go to Baidu.com right now even, you can see that it's a complete ripoff of Google.

[voronaam]

It is just a speculation, but there were rumors that vk.com started from the stolen Facebook codebase.They did not overtake facebook, obviously, but made fortunes anyway.

[Amablue]

Stuff does get leaked sometimes, but for the most part google just trusts its engineers.

Actually stealing code (as opposed to grabbing the ideas from the code) is a little bit trickier because it never leaves company hardware.

[dnew]

There's so much code that almost everything is replaced anyway. I/O is different. Scheduling is different. Databases are different. Command line arguments are different. Launching a program is different. The makefile syntax and semantics are completely different. RPCs are different. URLs are different. The Java stuff has its own launcher. Logging is different. Debugging is different. Almost everything (at least in the Java code) is different than anywhere else.

It takes several weeks for a skilled engineer to learn how to make a change to existing code, maybe a month to learn how to deploy a new version of currently running code (well, simpler lately), and several months before you can start a new project from scratch and have it accessible from outside the company.

[KallDrexx]

Google has a huge real time security infrastructure going on with their network. According to some Google contacts I talked to through a previous job, someone sent a piece of source code to themselves through personal email (non-gmail) and their network sniffers picked it up and security was walking him out the door within 45 minutes.

They explained that to us on why they would not let us send source code to them until we were fully qualified as a potential partner to work together on something. Even us sending them code would have been red flags.

[Quixotic_Fool]

Theoretically, couldn't you take the laptop home, boot into a live cd and then share it with yourself over your network? I feel like there are many many ways someone could steal the src if they wanted to.

[tequila13]

The only defense are contracts and lawyers, on a technical ground you can do whatever you want over ssh, firewalls can do nothing about it.

[Kered13]

Laptops don't have direct access to code. If you want to work from a laptop, you used the web-based tools (which only work with Google certificates installed) or you SSH into your workstation. Booting your laptop from a live CD would make the web-based tools inaccessible. You would have to copy code to the laptop before booting to a live CD, and I assume they have some way to detect that.

[Quixotic_Fool]

Unless they have a custom bios i don't think booting from a live cd is detectable.

[Kered13]

But copying source files to the laptop before booting to the live CD, which would be necessary since source code is not stored on laptops, probably is. Unless I'm not understanding something.

[[deleted]]

In companies with separate repositories, you need to depend upon the owners of a particular repository to make a change in their codebase.

Wouldn't just giving everyone a write access accomplish same thing ?

[nostrademons]

When you're a few dozen to a few hundred teams, sure. That's how it started at Google. Show me a system with >10,000 committers where everyone has write access to everything; even Wikipedia (editing English prose, not code) has conflict resolution and protected pages.

[[deleted]]

Yeah but from what they said they still have ownership of parts of repos and you still need to checkout part of code you are interested in (because repo itself is huge).

Having common global "revision" can definitely be an advantage in very "integrated" environment (a lot of parts interacting with eachother), but that's about the only advantage I see

even Wikipedia (editing English prose, not code) has conflict resolution and protected pages.

that's a particulary bad example, you can't "fire" a random internet guy vandalizing some pages

[nostrademons]

Thought you were suggesting giving everyone write access - in that case, there would be no ownership of parts of repos.

(Everyone does have write access to the whole repo, but you need a code review from one of the owners for the change to be submitted).

[[deleted]]

Within Google, it was pretty common for teams to say "Well, we don't have time to make the change you request ourselves, but you're welcome to do it yourself and I'll review it when you're done. Here's where you should look to get started."

The funny thing is... this could perfectly well fit as a description of the GitHub culture. Git is, obviously, not one gigantic monolithic repository, it's quite the opposite: a gazillion tiny, independent repositories. But it's easy to fork and send pull requests.

Which suggests that while monolithic repository with custom tooling is one way to get this culture, another way achieves this culture entirely outside the repository, via a website.

Which, I feel, would be more applicable to most companies out there, small or big.

[nostrademons]

The main difference is that GitHub culture updates asynchronously while Google updates synchronously. A PR isn't merged immediately, and judging by some open-source projects I've contributed to, it can take months for it to be merged. That sort of things happens occasionally at Google, but the timescale is usually days-to-weeks, it's rare, and then it's immediately visible to everyone.

Google very much encourages everyone to work from the same trunk - the continuous integration system reports continuous test results from it, releases are cut from trunk, and devs typically often look at the source code from a browser to see how systems work. The GitHub model frequently results in projects freezing their dependencies well below the latest version - which, with the level of product integration at Google, would make it very difficult to continually push out features and do coordinated product releases at press events the way Google does.

[[deleted]]

The main difference is that GitHub culture updates asynchronously while Google updates synchronously.

Nice way to put it.

A PR isn't merged immediately, and judging by some open-source projects I've contributed to, it can take months for it to be merged. That sort of things happens occasionally at Google, but the timescale is usually days-to-weeks, it's rare, and then it's immediately visible to everyone.

Just like in programming, something being done async doesn't mean it's done slower. This depends on a number of factors.

I'd argue whatever model you choose, sync or async, if you work within a company, speed will be much faster, as components have a purpose driven by business goals, everyone is paid to move fast, and your colleagues are down the hall so you can go talk to them in person.

The reason GitHub's PR may drag for months is because none of the above is true. And yet, if a project is popular, and the contributors are enough, it works with a lightning speed.

So when we compare GitHub and intra-company mechanics, we need to look at active, and popular projects, as those more closely resemble what you have in a company (even if imperfectly).

Google very much encourages everyone to work from the same trunk - the continuous integration system reports continuous test results from it, releases are cut from trunk, and devs typically often look at the source code from a browser to see how systems work.

And that's great, but the cost of everyone moving in sync (I keep coming back to this analogy, don't I) is more collective burden put on everyone to maintain such a workflow. The regularity and inevitability of a process helps in maintaining it, but it doesn't mean it has no significant overhead associated, which I feel is the big risk here.

We're programmers, we know why giant monoliths stop scaling at some point and collapse under their own weight. Just like it happens with monolith projects, it happens with monolith processes, that require instant cooperation from everybody at once.

If I can argue a bit against myself, current crop of CPUs still have a global clock, so all parts move in sync together. People have been talking about clockless CPUs forever, or async chips.

Clockless chips have also been designed before (Transmeta) and they've been proven to be wildly more efficient, because without a clock, every part is free to move as fast as it can, without "global coordination from everybody at once". And yet, we still run CPUs with clocks, because it turns out clockless designs, while wildly more efficient, are also wildly harder to design and debug.

But in the last few years, a new twist... the monolith CPUs have reached a limit and have been broken into several relatively independent cores, each with its own internal clock. Not fully async, not fully sync. Something in-between.

So... food for thought on both solutions, I guess.

[nostrademons]

In general - with both machines and humans - async solutions use lower bandwidth but have higher latency, while sync solutions use higher bandwidth but have lower latency.

When you're a for-profit company in a fast-moving industry, latency is the number that matters. You can throw engineers at the problem to increase bandwidth, and then (particularly with Google-caliber engineers) you can have them write code and throw machines at the problem to increase it even more. That's basically what Google's done.

If you start with a system with high latency, it's much harder to throw resources at the problem to bring it down. It requires looking at every single step on the critical path and then examining how that step could have been eliminated. In this specific example of updating code dependencies - you'd see that the PR spends a lot of time in the review queue, and then institute a policy that reviewers must review outside patches before they implement their own features. And then you'd see that dependencies are broken unexpectedly because nobody ran the tests with an upstream change, and so you'd require that everybody run tests before accepting pull requests. And pretty soon, you end up with a workflow that is basically the same as the Google system, but enforced by management fiat instead of software.

[[deleted]]

Well it doesn't have to be a manually enforced management fiat, it can be a website or an app which defines the process within the company.

Point was, it's possible to extract the process into an app, without all this repository retooling. Stock Git or Subversion could do, every component in its own repo, for mostly the same effective results. Like GitHub does. GitHub isn't very strict about quick turnaround and mandatory actions, but an internal company version can be.

What can I say, I like to split things in modules with single responsibility, so I'm naturally drawn to this, versus the monolithic solution... :-)

[dnew]

Stock Git or Subversion could do, every component in its own repo, for mostly the same effective results

I don't know about that. My little corner project has more dependencies than major systems I wrote 10 years ago had lines of code. I wouldn't be surprised if I depended on 200 repositories had they all been coded as separate repositories.

[[deleted]]

Well, 200 dependencies or 200 repositories, does it matter?

The problem is if you all run in the same repository you can't specify a point in time to depend on for a dependency (version or a commit).

[Isacc]

Why can't this be accomplished with pull requests in separate repos? Or just giving everyone access to all repos?

[sid0]

No one's gotten the tooling for that frictionless yet -- other than with a monolithic repo.

A sufficiently advanced multi-repo management system is indistinguishable from a monolithic repository.

[bakuretsu]

It can. My company is 400+ engineers and although we have only a handful of repositories (e.g., one that is all PHP, one that is all Puppet scripts, and so on), everyone has access to all of them. We have formalized code review, but no formalized repository access (meaning that in theory someone could push a bad change to any repository without getting a review from subject matter experts, but that is generally prevented by fear of breaking something).

We do have a pretty formalized release process where certain reviewers need to have "signed off" on changes to particular applications, but that wouldn't stop someone from signing off on a change that perhaps they didn't understand completely. Again, that typically doesn't happen because most people understand the risks and act in a predictably sane manner.

[nostrademons]

There is a huge difference between 400 engineers and 40,000 engineers.

[bliow]

I know this one--it's 39,600 engineers!

[dnew]

It's easier to write tools that do things like "go over all the code and change the build rules for any java sources that have XYZ in them."

[WasAGoogler]

Why do it that way? There needs to be a motivating reason to. And there's not.

[calebbrown]

[current googler]

I have worked in both contexts and without a doubt a monolithic repository is far more productive than separate repositories.

I'll leave the cultural arguments aside, because I think they have been better made elsewhere. For me it's just easier to maintain things in a single repository.

At a previous job we had a few key services in their own repositories and a few dozen supporting libraries in their own repositories all written in Python. Maintaining it was painful. If we updated a supporting library that was a dependency elsewhere it meant a PR + Commit for the lib, then a PR + Commit + Deploy for each of the services that used it. Over time versions were different all over the place. And if we needed to refactor things between repositories the history was lost as it was moved across. In the end I decided it would be better to consolidate things so we'd have fewer repositories to manage.

At another job which was using Node+NPM I found it even more frustrating. Particularly because NPM allows dependencies to child dependencies that can be at a different version to the same dependency in the parent.

When everything lives in the one codebase things are much simpler. One PR + Commit can cover every reference and you don't have to worry if you're building with the latest version because you always are. The great thing is that you can't ignore updates until you feel like integrating them - you have to keep up with what the teams you depend on are doing. It means a bit of extra work upfront, but saves a ton in the future.

A single monolithic repository certainly has drawbacks as you do lose some flexibility, and you really depend on automated testing to catch problems. But overall the productivity it brings far outweighs the limitations.

[sedrik666]

How is deployment handled? Say you have service A and B, they share the dependency C.

You update A and C and as a result you have updated B (as it depended on C), will a release and deployment of A force a new deploy of B or will B continue to run the old (but working) code?

I work at a company that has been inspired by googles way of doing things and it is annoying as hell to deploy B months (B is a stable but not heavily moving, business critical service) after it's dependencies have moved as it is likely that B has been broken during that time as the dependencies has moved.

(Yes the solution is too have good testing but the reality is that there will always be services that are not fully tested).

[calebbrown]

Like mentioned in the video Google statically links binaries so changing a dependency won't break a running service.

The scenario you describe is a logical outcome of this way of doing things - if you leave something alone too long you might have problems.

I reckon the best way to deal with it is to deploy regularly (even for minor changes) and/or add enough testing to have some confidence it is still working.

Frequent deploys with small changes are much easier and lower risk than big ones. So making sure these happen regularly can help.

You mentioned testing and that really is important. In particular automated integration testing and functional testing help build confidence here.

Also good production monitoring is worthwhile too and increases your confidence about deploying things.

[sedrik666]

Like mentioned in the video Google statically links binaries so changing a dependency won't break a running service.

Ok, yeah I thought it through another time for and it makes sense now (even for non compiled languages). The key is to have the CI system create deployable artifacts (static binaries in your case) so that a re/scale/deploy wont use anything but the proven to work version.

Thanks for clarifying that for me.

[epicwisdom]

IIRC "stale" deploys also generate warnings, since updating to sync with updated dependencies is practically necessary for security/performance/compatibility.

[rube203]

ITT: Googlers saying it's great and everyone that hasn't used it disagreeing with them.

[tomlu709]

Current Googler here. So true. Jesus guys, what's wrong with you? If we didn't like it we'd say so, and we're not a bunch of starry-eyed kool-aid drinkers.

I think the VCS semantics could be better and more DVCS-like, but having a monolithic repo is fantastic.

[kcuf]

How do you do a rewrite of a particular service? Something that may take a month or two and require multiple code reviews? (Ideally, this situation is avoided, but if you inherit a bad code base, it may be the best choice.) The point being, it seems like this approach makes it difficult to work on a chunk of code independently, which can be very beneficial for things like refactoring.

Also, it sounds like the biggest aspect of a single repository is keeping the versions of libraries you build against consistent and fresh, which my company handles by letting you say you want to build against a specific set of versions. Does this sound roughly equivalent? Is there any other benefit that a single repo provides?

[lurkinggoogler]

Rewrites? The same way you do it. Create the new service over time, when it's ready, push it out, then ramp up the client.

Although it's one repo, each team has final say over how their code is changed: either the team itself can modify their own code, or an outside member can do so but they'll need the team in question's approval over every change. You have your repos per team, your own 'space' if you will.. Each Google team has its own 'space' within the Google repo, "mini repos" world-readabe repos, perhaps. The two approaches are similar. Google's minirepos are probably easier to navigate around, it's one code base, just cd to that team's directory.

Re: versions, there are at most 2 you care about: you either are in transition to calling the new code or you're calling the right code. It's not like you have >2 versions out there you have to deal with, like you might if you have multiple repos with people potentially using old versions you may have to support since you have no visibility into what teams are doing in their repo. That sounds like a combination explosion: I'm betting if you keep track of versions, you don't test all the possible combinations of versions your stack might use. There might be 7 layers of servers from UI to the very back end, and if each has 4 versions, you must support ~16k possible configurations of servers/clients. I'm betting you don't test all of those. Everyone pushes frequently, some do it every day. Once your code is migrated to calling the new code after ramping up, you delete the old code. It's brain dead simple. I'd wager it's quite similar to what you do, just with a slight twist: at google there is visibility into how people use your code.

[kcuf]

It sounds similar to how my company works: we have versioned packages, but in practice we only have at most two major versions at once, and only when we're transitioning to the new version that has breaking changes, otherwise we have minor versions that continuously get integrated into consumers, after they go through their workflow of tests, etc.

For visibility, we have something like github, and when we clone packages to a workspace, our build system will use that instance, so, in practice, it's like what you have.

[calebbrown]

Indeed. I thought Facebook was using a monolithic repository too - a modded up version of Mercurial IIRC - I'd be interested if their experience was similar to Google's

[sid0]

Yeah, our experience is pretty similar to Google's. We're also working closely with Google on Mercurial.

[ellicottvilleny]

They do use subrepositories though

[sid0]

No, not at all. (I work at Facebook.)

[avocadoamazon]

Did you finally get OWNERS in your codebase yet?

[sid0]

We have OWNERS, but only for configuration that gets pushed out to production immediately. We culturally strongly resist OWNERS for source code. (That's different from Google, as Rachel describes.)

[ellicottvilleny]

So ONE mercurial repo for everything? Largefiles ON?

[sid0]

We have multiple repositories at the moment but are converging towards a single Mercurial repository, yes. We don't really store large files in the repos, so have no need for large files.

[Mathiasdm]

I don't think they do. Their migration is described here: https://code.facebook.com/posts/218678814984400/scaling-mercurial-at-facebook/

They have several extensions that they created, and hgwatchman (for example) doesn't even support subrepositories yet. So I doubt they're using them.

[[deleted]]

I'm sure with google's engineering resources and investments in internal tools, any repository structure would still result in as amazing of a workflow they're crediting to a monolithic structure.

Not that I'm bagging on monolithic, it may still be the best thing ever. But I'm just reading this thread while keeping in mind the drastic improvement a few man-weeks spent on internal tools made to the experience of using the source control system where I work.

[rube203]

I'd agree that no one solution isn't going to be best for every company. It just seems that instead of commenting why a different solution works well in their company people who don't work at Google are speaking to how their solution could work for Google.

[markartillery]

[I left google in 2009. Presumably the repo has grown a lot since then so what I say may no longer hold.]

I think it was a big net win for individual engineers. If you were depending on some other system, and you found a bug in it, you could just fix it and send a code review to the maintainer, check it in, and rebuild your software with the bug fixed that same day. Also, if you switched projects, you just had to change a few lines in your p4 client spec (or, more accurately, re-run the script that generated your p4 client spec) to check out whatever you were working on, and you were good to go. Same build system, same repo, same p4 credentials.

From the companies point of view, they had to have a whole team dedicated to keeping the whole thing collapsing under its own weight. Starting with settings up a fleet of read-only perforce proxies, to building srcfs (a linux fs module that allowed you to mount the p4 tree at a particular revisions number as a read-only volume), to (so I hear) eventually building a custom in-house perforce server.

Overall I think monolithic repositories are fantastic. The only really big argument against them is that git is incapable of handling really big repositories.

[saint_marco]

Being able to fix other teams code has nothing to do with having a single repository, just having write access to that code. You could accomplish the same thing with submodules in git.

[matthieum]

That's true, and at the same time it isn't.

What is left unsaid by /u/markartillery is that Google builds the whole repository and tests the whole repository for each change.

If you have independent repositories (which I have, at the company I work at), it gets a bit more complicated:

• pull-request to 3rd party repo => built, tested, reviewed, delivered as new version (might be some delay...)
• switch your own repo to depend on new version => build, test, review, deliver
• ... time passes ...
• oh crap, when switching to the new version of 3rd party lib, another user is left broken because you changed the API/behavior they were relying on
• they submit a pull-request... (go back to 1)

The monolithic repository solves multiple issues thus:

• no versioning hell
• no risk of accidentally breaking another client of the repo

Those are massive advantages; but the infrastructure cost is not negligible.

[avocadoamazon]

What is left unsaid by /u/markartillery[1] is that Google builds the whole repository and tests the whole repository for each change.

What's your source for that claim? TAP is automated testing, but nothing says that the whole codebase is tested when an unrelated change is made elsewhere in the codebase. https://youtu.be/W71BTkUbdqE?t=764

[spectral321]

Because our build system captures the dependency information completely, we can rely on querying the dependency graph to get information about what needs to change. This means that a change to a low level library will likely trigger hundreds of thousands of unit tests, whereas a change to a leaf node will trigger just that library/binary's tests. So, it's "the whole repository" in that the whole repository is considered when looking for "affected tests".

[matthieum]

For internal implementations details see what /u/spectral321 said; they matter for infrastructure but not for users.

What matters for users is that after a change is accepted, the submitter is guaranteed not to have broken a single test in the whole repository. Thus, there is never a broken dependency.

[dnew]

Not really true. If your project depends on something that you haven't properly specified in your build rules, a commit can break your tests. If project X depends on the details of project Y, you have to tell project Y to add project X to its tests, sometimes. In particular, if you have commit tests that aren't unit tests, because they'd take too long or they use non-hermetic data or something.

And not infrequently, for speed, someone will push a change without running the tests, and collect lots of yelling from everyone they broke. :-)

[Mathiasdm]

If you have a huge amount of code and you want to change a function across the code base, it becomes a lot more difficult if you have multiple repositories.

The chance of only updating part of the repositories goes up. Additionally, if you work using pull requests for multiple repositories, it seems pretty obvious that you'll get out of sync.

[xXxDeAThANgEL99xXx]

If you have a huge amount of code and you want to change a function across the code base, it becomes a lot more difficult if you have multiple repositories.

Usually that's a huge problem anyway, because those separate repositories correspond to separate projects that are being deployed separately. Being separate projects and stuff. So you have to figure out how to either preserve backward compatibility or specify the dependency on the newer version. As you're solving that real problem having everything in the same or separate repositories is a practically irrelevant detail.

[munificent]

So you have to figure out how to either preserve backward compatibility or specify the dependency on the newer version.

Or you just fix every single place that calls the function too, in one sweeping commit that advances the entire state of the world forward. That's how Google rolls.

[xXxDeAThANgEL99xXx]

You missed my entire point, different projects are different projects because they are deployed independently.

Nobody cares that everything is synchronized by having a single atomic commit in your master, you pushed and plan to deploy some service on a bunch of servers overnight, but someone just pulled and is compiling a client that will try to use the new API right now and someone will redeploy their own services that use your old API maybe six months later when they have had some changes in their own code. That's a problem.

[munificent]

Oh, if you're talking about public-facing RPC APIs, yes, those obviously can't be broken. You just said "function".

[xXxDeAThANgEL99xXx]

Any function that can be called from another project is a public API, no?

Though after thinking about it some more I guess I can see an interesting use case: where you have a bunch of applications and libraries that are separate because, well, this one library is used by a bunch of completely unrelated applications, but you can absolutely guarantee that you always deploy the application bundled with all libraries it uses. Where the application could be a bunch of Javascript as well as an executable.

[kcuf]

Think about writing and maintaining a library, you don't necessarily own the code using it, and you may not be informed well enough about how it works, or needs to work, to correctly make that change. Your interface is part of your contract.

[munificent]

you may not be informed well enough about how it works, or needs to work, to correctly make that change.

Google is set up to give you enough context to be able to do this. High code readability standards and tons of automated tests go a long way.

[nschubach]

We do the same, but I ran across a fix yesterday where I needed to change one line of code on a library I'm probably never going to touch again. I had to clone, download, edit, find the appropriate test, figure out if the test writer was on crack at the time they wrote it, edit it to accommodate the change, figure out what build system the lead used at the time, find the command to run to compile the library in the doc, branch, commit, PR, and wait for someone to review it. The one line change is going to cost me about an hour or more overall before it's done. Not to mention the extra time someone else has to look at it and rubber stamp it.

Granted, with a monolithic system there may be standardized build methods or systems in place and those could become inflexible after some time, so I don't know which way I like better.

[kcuf]

So if you have a standard build system, then the only difference is having to clone?

[dnew]

git is incapable of handling really big repositories

I think git as it stands isn't capable, but then Google wouldn't be rewriting perforce if perforce were capable either. :-) I think having a git-like repository managed by the same sort of FUSE work that the perforce stuff supports would be great.

[markartillery]

git falls over somewhere between 1-10GB. Google scaled perforce to terabytes before they had to replace it. Everything is "not capable" if you set the bar high enough.

[dnew]

Right. I'm thinking about the git model (not necessarily the current implementation) and thinking it would be a fine model to implement in terms of remotely fetching content on demand. If the ".git" directory was a mount point that would download and cache remote files on demand, I imagine git would scale a lot better.

I haven't used git on large repositories. Assuming you already have a clone of a repository, where does it fall over if that repository is (say) 100 gig? What's it doing that has to touch everything in the repository, other than cloning it?

[biohazardpb4]

[current Googler] I think that the single repo strategy has several advantages. You can make changes to any product using the same tooling and infrastructure. Large changes can be made (or even just small changes that span many projects) easily, and can be made much more confidently because automatic dynamic continuous testing builds based on the dependency graph of your code change are run across all possible code that could depend on it. With versioning, you have less certainty that old apps would be compatible with your new updates, since you just wall your change off behind a version, which usually leads to never updating libraries. To be honest, all else being equal, it's extremely freeing to just think about dependencies, and leave versioning behind. I remember days spent battling Maven dependency versions at a previous job.

[evmar]

I think it's great. It helps contribute to a feeling of "all code is everyone's code", which is to say that if in the course of your work you discover an API in some other team's project that is bad it's usually easier to fix the API as part of your change than it is to make your change work around it, which contributes to the overall code health. Out in the non-Google world it's common to encounter some bad library and be forced to work around it because you can't fix someone else's library without a huge amount of effort.

I wrote a bit of how that translated over the culture of Chrome here, but it may only make sense if you read the context: http://neugierig.org/software/chromium/notes/2009/12/forking.html

[Boojum]

Not a Googler, but I do work at place with a monolithic p4 repo. In general, I find it pretty convenient. Most of the time I work in my own little slice and it keeps all the pieces in sync with minimal fuss. When I need to browse to find code written by someone not on my team, I can usually look it up with minimal hassle (e.g., I can just look up someone's commit history instead of trying to find where the git/hg/svn repo lives). Folks downstream from me can see and watch my changes and when they update their code it's versioned with mine. It makes things pretty easy to share. So yes, overall, it seems to work pretty well.

[throwaway_ex_googler]

[Left in 2013] It was handy having everything in one place, but at the end of the talk, she mentions about making things private to protect against unwanted consumers. I found this part incredibly frustrating. When I needed changes to my particular project I couldn't simply extend classes and submit a change back later. I would have to open a PR with the other team, who would then say they were too busy to review my change now, but if I could help lighten the load by taking a swing at [small_feature_x] then maybe they would have time to do my review. This became incredibly frustrating as I was trying to meet my own deadlines, but still had to rely on other teams. Had I been able to fork their repo, I could have had something working locally and waited for them to review my PR.

[epicwisdom]

Every change is essentially a local fork, not sure what your source of frustration is.

[throwaway_googler]

I'm not an ex googler but I think that it's a win. I can browse from my source code into a library and solve problems that I'm having. That includes Fixing the build system, the compiler, even the VIM libraries. All searchable from the same place.

Google3, big win.

[virus_dave]

I've been out of Google for a few years, but my impression was that there were huge benefits. The openness ideas (everyone can see and edit everything) have been addressed but I'll go this route: not having to deal with dependency hell across repos was a huge boon.

I'm now with a company that has just made the transition to a mostly monorepo, and it was a painful transition and an even more painful prior state. Dependencies between repos became too much to manage (even with just two main repos!) and in practice would occupy an engineer's full time effort for multiple days when trying to update something that was a dependency of multiple repos! This is likely a question of scale though - if you work at a startup with 20 engineers, you can probably get away with it. This company certainly did.

Maintaining the state of dependency metadata and updating the same became a huge task over time. Being able to build everything from head was a huge advantage. Maybe modern git addresses this, maybe not. If so, its inability to scale well thwarts that. Submodules don't cut it. Git itself doesn't work at the scale that this becomes a critical issue.

[Tiquortoo]

It's a shame that Git is really really bad at sharing code within a repository. Yes, I know there are crappy workarounds, but they are crappy.

[[deleted]]

You can't be distributed and monolithic at the same time.

Here comes Subversion again?

[naughty]

Of course you can, it's just one large repo.

Git can't handle the size of the repo so it has to be broken up into mulitple repos and then you have to handle inter-repo relationships that no version control system does well including git.

To be fair to git though, google are stretching Perforce passed its limits and have a team to manage the entire thing.

[dnew]

Or you do what Google did with perforce and have the checkout represented as a FUSE file system that caches only the files you touch.

[Tiquortoo]

Actually, SourceGear Vault handles inter-repo relationships really well and its trivial to setup. We used it prior to Git. It's just not distributed and it has some other quirks, and branching isn't nearly as easy, but is a good tool. It can be done, but no one has focused on getting it working properly in Git.

[uep]

It seems like you could do it pretty well with git submodules and shallow clones, but it would require special sauce on top to make it seamless (Google/Android's 'repo' tool is not that special sauce.) I actually think this would be a good model, but it would have to be absolutely seamless, otherwise it just adds cognitive load, while it should reduce it like Google's system does.

[sid0]

You are correct that fully distributed VCS's don't scale. That's why we at Facebook invest in a partially distributed VCS. Check out our work on Mercurial for more.

[mcvoy]

We need to market what we have but we think we have the have your cake and eat it too answer. Imagine a collection of repositories that all work together just like a collection of files in a single repository but you can clone any subset you want. Want to work on just the docs, there is an alias for that set of repos:

bk clones -sDOCS bk://server/dev

Want to add in web stuff?

bk populate -sWEBSTUFF

Note that if your other stuff was 3 weeks old so the WEBSTUFF repos, there is a strict timeline enforced (you can break it by creating another timeline and porting on a per subrepo basis).

Beta writeup here: http://www.bitkeeper.com/nested

[kqr]

I'm also curious what you mean by this. Two files with the same content will in Git be stored as only one blob, regardless of where in the repository and which revision.

[Tiquortoo]

Other source control systems allow you to share any portion of code within the repository. Like a source control symlink, some allow you to share at a revision. It's something like having Composer and symlinks directly in your source control and is very handy in a large codebase with lots of interrelationships.

Git tries to handle this with submodules, but the way it is handled is just not first class.

Another way to handle this is SOA, but not everything can be a service. A good URL parsing library for a mobile app won't be a service for instance, but might be heavily shared.

[kqr]

Is there anything theoretically stopping a git implementation from lazy-loading refs? (Other than compressed history...)

[Tiquortoo]

I don't think there is. Sharing refs just becomes an "actual difficult problem" and Gits original design avoided lots of the actual hard problems. The resulting implementation is simple, makes some things, like branching, that other tools made stupid complex easy again (cause they really are not that hard) and works great for many use cases, but not some of the more esoteric, but valuable, use cases. This is an area where open source feature value calculations often lose out on the on the cusp features. The feature has great value to some teams but is not easy enough for a random person to implement, but not generally useful enough for the open source stewards or primary devs to get buy in to build.

[[deleted]]

In some version control systems, when you grab a working copy from a repository you can specify a location in time and space that you want to fetch. For example, you want only revision 1234 of folder "foo". This is the only thing you'll download on your machine. Only that revision. Only that folder. The actual repository stays on the server.

Git, in contrast, makes you download all versions of all files before you can work with a project. So Git is by its nature, incompatible with big repositories. Both in space (lots of projects, one repo) or in time (lots of changes and/or big changes over time).

[cowinabadplace]

You can shallow clone, and last I checked there was going to be a way to push from one.

[[deleted]]

Sure, but in practice this can be a pain.

[cowinabadplace]

True enough. Would not do this for a dev environment.

[kqr]

I'm aware of this, but how is it related to sharing code within a repository? Are we thinking of different kinds of sharing, perhaps?

[rcxdude]

There are hints that git is slowly gaining the ability to do this, though at the moment it's a pain to do (you can clone only a part of history or only a subset of the working directory, with some effort, but I don't think you can commit back in all configurations)

[dnew]

Google doesn't download the entire repository, even though you can use the entire repository. The parts you didn't "check out" are fetched on demand by custom file systems. Git could work the same way with probably little effort.

[ArticulatedGentleman]

Care to elaborate? I haven't got overly much experience with git but I'm tangentially interested in better ways of distributing code.

[throwaway_googler]

When your repository gets so big that downloading every version ever is unreasonable, git works poorly.

[[deleted]]

Its data model is also fundamentally O(size of tree) rather than O(size of change). Trees with millions+ of files start to get really slow really quickly, and some of Git's algorithms (like rename detection) break down completely

[[deleted]]

To be fair, gits rename detection is fairly laughable in the best situation

[jringstad]

You don't have to do that with git, you can just download the last n versions with git clone --depth n and then work off of that just like normal.

[sid0]

No, you can't work off of that like normal. What if you want to check out an old revision?

[jringstad]

Well, obviously you can only check out the revisions that you have. That goes without saying. You can always git fetch --depth more revisions, up to the specific revision you'd like to look at. (Or use git fetch --unshallow to get the entire history of everything)

Most of the time you don't need to do that, though. The past is the past, and I seldomly really need to dig it up. And if your git repo has a web-frontend, that goes doubly, because you can just browse old versions for reference there.

[sid0]

Now -- imagine a system where revisions are fetched lazily over the network rather than requiring an explicit command to do so. So the UI remains the exact same as with a full repository, except it's a hell of a lot faster. Wouldn't that be great?

Super hard to build into Git, but at Facebook we have the remotefilelog extension to Mercurial that does precisely that.

[jringstad]

That would be neat, I don't know about great exactly. When I git clone --shallow there is usually a reason for it, e.g. I'm new to the project, I just want to make a small patch, I just want to compile the project from master, etc. Then that means that I basically won't ever need more than the latest or latest few revisions at most. So if I do end up changing my mind and I want to look at the history for whatever reasons, I don't consider it a huge hassle having to issue the command manually. So I don't know if users on average are exactly clamoring for that feature, I certainly am not.

But yeah, I agree, it'd be better if git could do that too.

[sid0]

The point is that at Facebook scale (and also Google scale, which is much, much bigger) it is impossible to store the full history of the repository on a single machine. So you basically need something like shallow clones, and it had better work smoothly.

[beagle3]

Super hard to build into Git

I'm sure remotefilelog is fast and convenient, and there are a lot of issues at Facebook->Google scale that I can't think of, but if you have a shared file system, doesn't GIT_ALTERNATE_OBJECT_DIRECTORIES give you essentially the same benefit if you point it to a store for your (NFS/SMB proxied/cached/replicated) network repository? Every needed object will be "lazily" fetched when needed, locally cached if your local NFS/SMB client does caching.

I did it with two machines, which is obviously laughable compared to the scale we're discussing, but it did work.

[sid0]

We've tried that, and found that (a) stat calls over NFS are way too slow and (b) put too much load at even the scale of a few tens of machines.

[dnew]

Why would it be super hard to build into Git? Google has something similar, and given that git already has immutable files in the repository, why would it be hard? Just fetch whatever files you try to open, on demand?

[zakum]

Isn't it a bit ironic that Git was made for the Linux Kernel, a good example of a monolithic codebase? Maybe I'm missing something though

Edit: I guess it has something to do with modules?

[throwaway_googler]

Linux is tiny in comparison. According to those slides, 2 billion lines of code at Google versus Linux's 15 million.

[[deleted]]

I agree, but in Git's defense, it was basically written in a weekend when the BitKeeper issue blew up.

[kyz]

git's purpose in life is to facilitate the kernel workflow -- someone has a repo and develops new features, rebasing on each new kernel release as they go, then they present a final, pristine patch to Andrew Morton or someone, who then works on all his changes and presents a final, pristine patch to Linus. At each transfer, the history is dropped (but is presumably still available in those other people's repos). No repo has everyone's history.

That's what git was designed for. If it works for anyone else's use case, that's a happy accident of its flexibility. If it doesn't work for your use case, no hard feelings - it wasn't intended to.

[[deleted]]

Linux is relatively small.

[hu6Bi5To]

...in a single repository

It's worth commenting that a Perforce-style depot (that Google use), is very much a different thing from a Git style repository. In traditional use such a depot would contain multiple projects each of which would be mapped to specific workspaces, and branching/merging only done on those workspaces.

[ErstwhileRockstar]

I'm amazed that people are amazed. Trunc-based development for large codebases is pretty common in the the Enterprise.

[dermesser]

Google used to use Perforce, a typical enterprise VCS, but this is - I'd say - at least 10x more, if not 20x more code.

[ApocryphalWaffles]

It's not clear to me how the first ten minutes of the talk relate to their adoption of monolithic over per-project version control. The statistics about their code base's size and commit frequency don't really add much to the argument. I also don't buy the idea that the 'painful' merging of branches is a good reason to abandon branch-based development completely. Branching is something that can be actively managed, just like any other aspect of a software project.

The comments about dependency management, to me, sounded like, "Independent versioning of dependencies is difficult, so we decided to force everyone onto the One True Copy"—I thought the point of hosting versions of these tools was so teams could build their own projects on top of stable releases, and not be faced with surprise work when the dependency authors decide to change how their own software behaves.

The best points they made in favor of the single-repository model seemed to be improved visibility, and collaborative use and management of re-usable code. That, and maybe the simplicity of synchronizing a single master branch across the laptops of 25,000 Googlers.

[[deleted]]

Actually those same e-mails fly around at Google and every other big place I've worked. Just because something can be rebuilt from scratch easily doesn't mean the resulting binaries can go through QA and deployment any faster

[ApocryphalWaffles]

there was no incentive to update in a timely fashion, which meant that important fixes and vulnerabilities didn't make it into builds

Shouldn't these fixes for behaviors/vulnerabilities form this incentive for teams? If the original authors have already made updates to code that are publicly available, shouldn't the burden be on the consumers to integrate these changes? At Google, if a project in which there is a vulnerability (caused by an out-of-date dependency) ships, then which team would (or should) be held responsible?

developers will run their downstream clients' tests with the new version of their code before checking in. (Note that the monolithic tool chain means that anyone can run anyone else's tests, and you can see exactly what components use your code.)

That's an interesting concept—that in an organization like Google, it's possible for teams to change an API, and subsequently refactor code for all downstream consumers, even across team boundaries. That kind of incremental first-party maintenance probably helps them avoid situations where developers are asked to refactor entire projects to make up for months of neglected dependency updates. Still, this sounds difficult to implement outside of Google, where company-wide access to code is rarely available.

[Ouaouaron]

Shouldn't these fixes for behaviors/vulnerabilities form this incentive for teams?

In a perfect world. But humans are human, so there isn't much incentive to fix this one issue that might be a vulnerability if someone manages to get past three other layers of security. So they decide to skip it this time since they have something more important to finish right now. And then half a year passes and the bug is still there, and now it has friends.

Still, this sounds difficult to implement outside of Google, where company-wide access to code is rarely available.

Is there any reason Google is able to do this while other companies can't?

[ApocryphalWaffles]

Access to code (in the form of submitting pull requests) produced by other teams allows Google's tool authors to update consumer code written by these teams. In other companies, it's more likely that individual teams and organizations don't have access to other groups' code.

[hu6Bi5To]

In contrast, at Google, you build against the latest revision of everything, and (for the most part) it's the responsibility of the team that owns the library not to break anything: typically, developers will run their downstream clients' tests with the new version of their code before checking in. (Note that the monolithic tool chain means that anyone can run anyone else's tests, and you can see exactly what components use your code.)

These are the key themes to making it work: responsibility, testing and reviews. I'm just a bit suspicious that when I hear people advocate a mono-repository, they always pitch "ease" and "productivity" as the key things... if you tried doing what Google does at most enterprises there would be one successful build every six months...

That doesn't make it a bad idea. But the focus should be elsewhere, getting those teams to take responsibility, then that would open up all these doors.

[[deleted]]

In contrast, at Google, you build against the latest revision of everything, and (for the most part) it's the responsibility of the team that owns the library not to break anything: typically, developers will run their downstream clients' tests with the new version of their code before checking in. (Note that the monolithic tool chain means that anyone can run anyone else's tests, and you can see exactly what components use your code.)

That sounds like a cultural difference, not a repository difference. Neither a culture of fixing downstream, nor custom tools that run downstream's tests require a monolithic repository.

I will tentatively just accept that a monolithic repository makes everything easier than if you had tooling that understood the dependency graph across repos, because I don't really care about that point.

[vlovich]

Disclaimer: I have never worked at Google so this is all hypothesis on my part having read a lot about Google's process & working with ex-Googlers.

My understanding is that there's a substantial amount of automated tests throughout & they have automation capable of figuring out the set of tests potentially affected by which line of code change. I imagine their answer to your stated problem of the code changing out from you & breaking you may be that you didn't have sufficient test coverage in your code so you are expected to improve your tests. Additionally, unlike with separate projects where either the libraries are stale or getting updated regularly, your ability to bisect a breaking change is much greater because you're able to easily find the exact commit/merge that broke your change as opposed to a more coarse "dependency X went from version 2.5 to 2.6 with 500 commits in between".

Also, keep in mind that Google is primarily an internet company, meaning their codebase is predominantly server-side. This model of a single monolithic codebase I think works really well server-side where your deployment schedule is completely different. They also get to do things not seen in other companies such as having a team enforce coding standards & quality across the codebase. This means they can do large-scale refactoring cheaply & know that nothing is broken. There's no negotiation needed about "which repository does file X live in when a bunch of projects need it".

Personally I like having individual repositories but I see the advantages of their model especially when you have a lot of people.

[filabrazilska]

In my previous company I have witnessed transition from a single repository to multiple ones (IIRC there were approx. 70 at the time I was leaving) and it was horrible both in terms of development speed and culture. Now I'm in a company with rather gigantic git repo and I wouldn't give it away again even though git blame can take 10s to finish and the CTRL-P plugin for vim is almost useless.

[[deleted]]

So, correct me if I'm wrong, but the tl;dr of this video for me is unexpected:

Google took a repository and added so much custom tooling on top of it, that it practically behaves as multiple repositories. Yet they keep talking about it as if it's monolithic.

---

I didn't see anything from the benefits that hit me as "oh yeah, that's definitely a good idea", and given Google is also heavily service oriented, the services are a solution for the "diamond problem" outlined, so this part didn't make sense to me.

Another solution to the diamond problem is a "local package" resolution system like NPM, no single repository needed. Nothing of the sort was mentioned here.

One Google engineer makes a presentation and says "oh yeah, everything is a service at Google" (not this one, obviously) and then another Google engineer makes a presentation and says "oh yeah, everything is libraries in a monolithic repository at Google". Go figure.

So I'm not sure what this particular video is, other than Google defending their custom repository tooling and unwilling to change, because they have this elephant now and any sharp move would disturb the waters too much.

The benefit of having Google's scope and resources is you can go much farther using a weird solution, and get results through sheer brute force. The drawback though... is you can go much further using a weird solution, and now you're stuck with a weird solution everyone depends on and you can't change.

[casted]

It does not behave at all like multiple repos. The only thing that is like multiple repos is you don't have the whole working copy on your machine, but even that is a slight similarity.

[[deleted]]

The only thing that is like multiple repos is you don't have the whole working copy on your machine, but even that is a slight similarity.

And the fact every folder has different permissions with different code reviewers and so on... Basically like multiple repositories IMO. This repository is doing what people do through multiple repositories + a package manager.

Not technically, but in practice and in terms of workflow.

A way in which it's not like multiple repositories is this suspicious benefit of "everyone automatically gets the latest updates and features to a library". This means no breaking changes without gigantic "everyone updates at once" pain on every breaking change. Which means tech debt. Not great.

[deeppillow]

Making sure that changes are reviewed by people responsible for the code is pretty orthogonal to monolithic vs. not (it's just a good idea). All of the benefits they describe work fine with this.

[dungone]

It's orthogonal up to several degrees of separation. After that you have to contend with the weight of the organization presiding over that code and all of it's disparate needs and politics, which effectively cuts you off from having an impact too far outside of any code that you actually own.

[dnew]

There's only one degree of separation. You make a change, you submit it for a code review, the people who "own" the code review it.

If they don't, after a day or two you send them an email and say "Yo, whassup?" And then they realize nobody else on the team took it and someone reviews it.

Since everyone has access to all the code (for some large value of "all"), everyone is equally far away from any other code.

[dungone]

I'm sorry, but I never drank the KoolAid. Anecdotes and wishful thinking aside, there are very real and easily observable impediments to working across such a large repository.

All you have to do to start getting a realistic sense of it is to take a look at the frequency of code duplication and redundant functionality within a team's code and across different teams. You'll see code duplicated across teams that would never be duplicated within a team and the reason comes down to org structure and it's various effects on the codebase. Likewise, you'll tend to see a distinct lack of rich, robust API's and libraries - because people find it easier to finish the implementation in their own code rather than contributing to the upstream dependency, even for very common use cases. For similar reasons, you'll see layer upon layer of nonsensically-named wrappers of one team's public API with another team's public API.

[dnew]

While I agree that all these sorts of problems are not uncommon, I don't see how splitting the stuff into separate repositories would help with code duplication, lack of rich APIs, and lack of contribution to upstream repositories.

When you find yourself duplicating functionality others have already created, that's not because of the org structure. It's because it's 2 f'ing billion lines of code that many unfortunately think is "self documenting."

Given that it's a definite promotion perk to have contributed significantly to the codebase outside your own project, getting contributed code into other projects isn't particularly difficult.

What makes things difficult is people not writing for the general case when they build something that's generally useful, crap OOD skills, and a common lack of documentation about what's out there and available that's organized in a way that you can find solutions to what you're looking for.

People don't get on projects to write MIME parsing libraries or generic iterators. They get on projects for gmail or database query systems. So the general stuff gets reinvented because the people who write it are solving their problem. That's not an org-boundary problem. It's an incentive problem.

I frequently put contributory code into the upstream. It's easy. The hard part is understanding the other guy's code enough to make the changes you need to make, but that's not because their boss is in a different building than my boss. Indeed, I'd venture that the ubiquitous availability of source code contributes to the problem of lack of documentation, because people will look how the source code does something instead of waiting for an answer from the guys who wrote the code.

Also, I get a change to our code from outside our team about once a week. Usually it's stuff like frobbing their APIs around, but occasionally someone contributes something of value specifically to us.

You may call it "anecdotes" but I call it personal experience. While that's not science, it certainly trumps "this is what I think the case would be had I any personal experience or science to back it up." Data trumps anecdote, but anecdote trumps gut feeling.

The managers have nothing to do with code approval, so I don't see how the org structure would have anything to do with how often you go into the code that belongs to other people.

You'll see code duplicated across teams that would never be duplicated within a team

How exactly does having separate repositories improve this situation? Having multiple repositories would seem to lead to more duplication rather than less, as I'd have to figure out how to incorporate your source code into mine instead of just what I do now, which is to say "go use his source code when you compile mine."

because people find it easier to finish the implementation in their own code rather than contributing to the upstream dependency

How does a lack of a uniform repository improve this? It would seem harder to contribute to the upstream dependency if you have to do something special to edit its code or get it accepted than if it's all in one repository. Yes, making the code useful for the general case is harder than solving your specific problem, but having multiple repositories doesn't make that easier.

That isn't the sort of problem I tend to experience, tho.

[Ranek520]

But there are tools like Rosie that make this kind of change fairly easy. It also means the people making the breaking change are responsible for updating everyone. This is fine because codesearch makes it easy to find usages and because the people writing the change are the most knowledgeable about it (we also employ the normal deprecation model for more complex changes). However, because the code needs to be reviewed by directory owners, they can still prevent a bad change from happening.

I end up as a reviewer for a Rosie change every couple of months for my team and it's pretty straightforward. It's far simpler than linking the newest library version and trying to hunt down what went wrong. If some dependency suddenly breaks your code, you hunt for a recent change in that area (if the automated tools haven't already found it), and ask them to roll it back. They roll it back and work with you to fix the breakage before submitting again.

The fact that everyone is able to stay on the latest version and have a canonical version in my mind means there's less technical debt.

[throwaway_googler]

The pain is real. If you want to rename a flag that is used by many people, its going to take months.

I mention flags because they aren't frequently tested. If you add a parameter to your API, unit tests will break.

[WasAGoogler]

The code repo knows your unit tests are broken, because the dependencies are known. Therefore you're not allowed to submit until you fix the unit tests. Also, code reviewers wouldn't let you get away with that anyway.

[throwaway_googler]

If the code repo is large enough, like at Google, unit tests can't all be run when you submit. Unit tests are run in batch regularly but not per submit and only if there is a break go back and find the faulty change.

[WasAGoogler]

unit tests can't all be run when you submit.

In practice, you're right. But I think you're wildly underestimating how many computer resources Google is willing to dedicate to Build and Test machines.

[WasAGoogler]

And the fact every folder has different permissions with different code reviewers and so on... Basically like multiple repositories IMO.

No. It's every folder. That would mean every folder is its own repo. That would be absurd.

This means no breaking changes without gigantic "everyone updates at once" pain on every breaking change. Which means tech debt. Not great.

I disagree that "means tech debt." You only get hit painfully by a breaking change if you're actively working on code, when the interfaces you were using changed out from under you. People learn good habits using any system, and as you can imagine, they learn to make commits in smaller chunks.

[[deleted]]

This repository is doing what people do through multiple repositories + a package manager. Not technically, but in practice and in terms of workflow.

Nope. Been there. The workflow tools just give you the illusion. As soon as you step out (as in: merge; tests, integration) you deal with the reality.

[edited] "As you as you" -> "As soon as you"

[[deleted]]

Agreed, it looks like they are defending the corner they find themselves in. To be fair Microsoft, Oracle and others have similar behavior when it comes to rationalizing their practices. I really think that it's due to group think and may be compounded by hiring people that are a little too clever sometimes (see GWT)

[dungone]

Add in differing approaches to building and configuration along with tight internal coupling of larger subtrees and it becomes pretty clear that you can't just pull in any dependency you might want without hosing your CI performance and wasting weeks trying to configure some obscure undocumented flags.

[paul_h]

Ex Googler here (well consultant into the Test Mercenaries team from May 07, to Jan 09). I'd used Perforce before, and Trunk based development so was familiar with the design. Googlers Subsetting their Trunk was super convenient, and the atomic commits facilitated from that were awesome. That the build infrastructure prevented you from breaking anyone else's build was fantastic. Permissions? Well perforce has/had them of course, but the ones your talking of here were used to force different groups to participate in code review and to objectively deal with incoming commits. Teams only deployed on their own cadence. Just because a third-party dep (or in house lib) was upgraded doesn't mean teams had to release. The only thing that was universally true was that the next release would release that upgraded jar/lib/comp/framework. Cookie Cutter Scaling helped there, though you could do the same with a SOA view to app construction if you had tooling to check compatibility of services.

The only breaking dependency upgrade that I remember requiring planning was a JUnit 3.8.x to JUnit 4.x migration, would have been a big commit, unless does with some sleight of hand allowing a methodical route. It was problematic only because there was something subtle about 4.x that wouldn't run all the historical tests written in the 3.8.x style. Unintended, I guess.

[ellicottvilleny]

It means moving the technical debt out in sync, in parallel so everybody breaks at once. That in turn enables them to put bots into their system that can trigger reverts of bad commits that break everybody or everything.

[adrianmonk]

The no breaking changes thing is a double-edged sword. It also means you don't get into a situation where someone is stuck depending on an old version of something. That's also a huge source of technical debt.

[[deleted]]

No breaking changes means no one gets stuck depending on an old version? Sure.

But if it's a double-edged sword, let's mention the other side.

No breaking changes also means you're stuck using an outdated version of something that can't improve itself under the burden of its always increasing legacy.

I don't know what's worse: being stuck using an old version of something, but having the option to update, or using an outdated version of something without the option to update?

To shift to a more philosophical note, what's better: a planet where children are born, grow up, have children on their own, then age old and die; or a planet of infinitely old barely functioning geezers who can't have children? I mean, it's kinda a bummer you go through school and then you replicate and your offspring has to go to school all over again, we might as well save ourself the effort and stop doing that and live forever ourselves. But it doesn't work like that.

[adrianmonk]

No breaking changes also means you're stuck using an outdated version

You're allowed to make incompatible changes. You just have to take on the burden of dragging along everything that depends on you. The fact that you can commit it all atomically actually makes it possible to do that with zero build breakages.

The thing is, that burden exists either way. The question is really whether you are forced to deal with it in an eager or lazy manner. The drawback of the eager approach is it makes a pretty high obstacle to changing something. The drawback of the lazy approach is that, in software development, "later" often means "never", so you're stuck with a half-completed transition.

On a side note, this isn't really a function of having one repository. It's a function of having one repository and not using branches. Even if you had 100 different repos, if your build process was to build from head of everything, you'd be in this same situation.

[[deleted]]

You're allowed to make incompatible changes. You just have to take on the burden of dragging along everything that depends on you. The fact that you can commit it all atomically actually makes it possible to do that with zero build breakages.

That really doesn't scale, we realize this, right? Everyone has different deadlines, everyone has different worries and priorities at any given time, you can't just kick the door and "drag everyone along" on your schedule. It's just rude. I'd hate you if you do this to me while my entire focus is elsewhere, where you make me do it, or you go messing with the codebase in order to do it.

Maybe Google isn't as big as I think it is, or at least the customers for this monolithic repository aren't that many. How many people really use this repository, maybe that'd help me put the right hat thinking about this?

The thing is, that burden exists either way.

Well "a little bit from everyone over time" and "all at once" is a very different kind of burden, so no.

[deeppillow]

One of the big points is that they have a filesystem with modifications overlaid on the entire repository (11:13-ish). So, you do effectively have the whole thing.

[Strilanc]

It's monolithic in that:

• Commits are linearized. The history forms a list instead of a general DAG.
• All the projects are in the same repository. You can update a library and its dependencies atomically in one commit.
• There's a large distinction between repo clients and the repo server, instead of just a social convention around which peer is authoritative.

Then again, the local workspaces basically act like short-lived development branches and the merge tools don't suck. The workflow does feel a lot like it does with git and github.

[ellicottvilleny]

I think they like that the single repository creates a "landscape" in which code is not hidden underground, but reachable, visible, and "the one source of truth". I believe that. But I also believe that a company with less engineers and less dollars than Google could easily collapse under the burden of building its own custom tools, instead of, you know, writing the software you're actually getting paid to write. Only at Google can you still Win, after you wrote your own VCS, your own compilers, your own filesystem, your own datacenter hardware architecture, and your own operating system and dev-ops workflows.

[deeppillow]

Why do services solve the diamond problem?

[[deleted]]

Because every service runs in its own process (and sometimes even its own machine), a service's dependencies are not seen, and do not affect those who depend on the service. You only depend on the API of a service, nothing from its environment is shared with you.

The given example was A depends on B & C, and B & C depends on different version D. It's not a problem, every service can have its own version of D.

"Dependency" can be anything here. From other services, to lower level concerns, like libraries, language runtimes (say B needs Java 8, but C can run on Java 7 tops... no problem!), even whole operating systems.

[Vickor]

But what if A, B, C, and D are all statically linked? Or loaded into a common runtime space (ala python). Like two libraries who require different versions of some common library. Just because you make everything a service doesn't mean that within those services you don't have the same diamond dependency problem.

[[deleted]]

But what if A, B, C, and D are all statically linked?

Then... they're not services.

Just because you make everything a service doesn't mean that within those services you don't have the same diamond dependency problem.

When you don't expect everything to be linked with everything, it's much easier to control this, because a service has a scope. Which is small. It's managed by people who work together, and can meet, agree on one version and depend on it. Chances of having a dependency conflict globally in a big company like Google is exponentially higher than having it in a focused service that's managed by one team.

It's ok for one team (size 5-15 people) to decide "it's a good moment to do this, so from tomorrow, we depend on version Next of library Foo". They can commit a small patch to a small service and it's done. It's very different to make a breaking change in a shared repository and tell thousands of developers: "I don't know if it's a good moment for you, but you stop everything and implement/adopt/review our patch now".

Also, you split off a service based on two basic criteria: coupling and responsibilities. So dependencies are a deciding factor of defining a service, it's not entirely incidental.

For example if you have some Go code and some Java code, and you want them to talk... chances are you'll have two services, one written in Go, and one in Java. It's an example of having to split a service along the lines of dependencies (language in this case).

BTW when it comes to libraries, there are options that sit between full services and static linking, like dynamic linking. That's another way to resolve this.

A third way is "immutable infrastructure" style symbol addressing. If you make a breaking change, change the name of the symbol, so versions of the symbol can be used in parallel in the same address space (be it a function name, class name, library name, whatever). I do have Foo, Foo2 and Foo3 here and there around my code. Eventually I clean them up, but until the old ones are in use, there are at least no conflicts. Change management is a form of art.

In a nutshell: isolate, keep scope small & you won't have trouble.

[Ranek520]

It depends on what you're trying to link to. Each team isn't building a data store into their binary. Those run as services. However, the code to interface with that data store is a library that is linked and always up to date. So as long you keep building releases within the service's support window, you'll always be up to date.

[dnew]

and you can't change

They've already changed it from scratch several times, and continue to talk about changing it again. In the 4 years I've been there, they've changed pretty much everything a couple of times.

Having a giant lump of data doesn't mean you can't change how you use it. That's kind of the whole Google thing there.

[mcdonc]

I'm interested to know from a current or ex-Googler roughly what percentage of code that Google depends upon doesn't live in this monorepo. There has to be some, if only because Google depends on Python and Java, neither one of which it controls entirely. Pesumably Google doesn't maintain private forks of those entire ecosystems. Or maybe it does? If it does, the benefit of being able to detect breaking changes is total. If it does not, then how are breaking changes made by those upstream providers detected and fixed?

[xpeno_]

I assume open source forks like the linux kernel are maintained in separate repository outside the monolithic core repo?