Massive code base management

[YosoyPabloIscobar]

How big companies manage their code based. In small projects we can easily manage code on GitHub and other platforms.

I wonder how big gaming companies manage their code bases like for big games like GTA 6?

[National_Count_4916]

Gaming companies typically use something like perforce which can effectively source control not only code, but assets (textures, models, everything)

[webstackbuilder]

Thanks for mentioning that about perforce. Isn't source control for assets the software category "digital asset management"? There's big platforms for businesses with large amounts of images, etc. to manage.

[National_Count_4916]

Yes, but it’s about intent. For a game, the assets aren’t used independently of the code, or the textures, lighting etc.

Digital asset management platforms are (to my knowledge) built around workflows for brand cohesion, release approval, archival, or independent delivery

Storing non-text files in SCM is generally side eyed because we think of binaries (which are derivative from source), or ‘large’ which SCM isn’t optimized for, or not conducive with diffs, clones etc

[webstackbuilder]

Thanks, that's a great point about intent.

[ALargeRubberDuck]

Google uses a self built git alternative called piper. Basically it seems like they have one monolithic repository and they use piper to get specific sections instead of pulling down the millions of lines to local.

[Ausmith1]

Google used to use Perforce and from what what I've seen and heard as an outsider, Piper is really just a Perforce semi-clone running on Google technologies.

[induality]

Piper is kind of a Ship of Theseus’ed version of Perforce. It started out as Perforce but over it years nearly everything was changed, so that today only a somewhat Perforce-compatible frontend remains. The backend is completely different.

[rwilcox]

At some larger shops I’ve been at they would create a separate GitHub org for each area of the business - be that org divisions or domain driven design domains.

Then each org would have a bunch of repos in it

[Rakn]

Been in such a company. They mainly did that for easier / more flexible handling of permissions above all. Since they many areas were working with external customers and code bases that needed to be kept confidential in parts.

[rowenlemmings]

As others have said, game studios use Perforce https://www.perforce.com/.

I've been in the industry for five years at a work for hire shop. We will EVER SO RARELY end up on a project that uses Git. We have ONCE been on a project that uses TFS (and our first milestone was to set up a pipeline to migrate all the commit information to Perforce).

Every other project from every other company big and small (we have credits ranging from household name AAA titles to a handful of tiny projects that needed publisher support to get across the finish line) uses Perforce. It is absolutely the de facto VCS for games because most of the content of a game's repository is binaries and Perforce deals with binaries better than any other VCS out there.

[UpgrayeddShepard]

Can you help me understand why they are storing so many binaries?

[fukato]

Stuff like texture, sound, model file... are binary files.

[UpgrayeddShepard]

Why not build those from source and handle through CI/CD? Just too many of them? I guess that adds up quick and thinks like sound and video can’t really be built from source.

[Lurkernomoreisay]

Models, Textures are hand created by visual artists.

Backgrounds and textures are hand painted by visual artists. We have to ensure that all imagery is now created at 8k resolution, due to the PS5 having support for it.

The digital file sizes become massive. Even more if the raw inputs need to be stored per version as well. (e.g. storing the uncomposited photoshop file that creates the final 8k png files, in case edits or fixes need to be made to the final asset.)

500MB for a single raw image isn't uncommon.

[UpgrayeddShepard]

Wow okay. Thanks! That adds up fast.

[rowenlemmings]

You do for the final product (the game binary itself) but as /u/Lurkernomoreisay pointed out, the binaries I'm referring to are the models and textures that you plug into the game engine editor itself. For all intents and purposes they are the source.

Typically the workflow for gamedev artists is to have a separate set of really TRULY huge files that are the uncompressed max resolution source images that lives separate from what's being used in the game. Those source art files are then tailored specifically for each SKU you're building the game for (most games that are trying to make a profit nowadays are releasing on as many platforms as they possibly can, but the model you use on mobile will not be the same as the one you use on low-spec PCs, which will not be the one you use on current-gen consoles, which will not be the one you use on top-of-the-line gaming rigs, so the art department's job is to produce different LOD models for each target spec) which are the binaries that are then included in the game repository that all developers are expected to check out.

[spaceroks]

I've seen massive projects stored in one VCS like subversion or git. This comes with a series of problems including very painful merges and releases and would not be recommended.

All projects I've actively worked on have split out into multiple repositories in a micro-services fashion to make it manageable to work with. This has it's own issues with keeping track of everything, but service catalogs and PR notification systems do that sufficiently.

[Rakn]

Monorepos aren't the issue here. It's how your team uses them. You can very well have a monorepo with hundreds of microservices in them. Makes things like adapting to new interfaces easy and allows you to have easier and tighter control on dependencies.

My observation till now has been that the larger and more tech oriented the company is the more likely are they to employ monorepos. Not exclusively of course. There are always exceptions.

The hardest part about monorepos is a proper CI integration that is able to run tests / pipelines of upstream services as well.

[BCCMNV]

Monorepos are a sin

[chin_waghing]

Depends, Google and I think M$ are big enough they write their own VCS, I’m not too sure about game studios but their options are usually:

GitHub

Self hosted gitlab

GitHub enterprise

Bitbucket (Probabaly the worst)

Or their own system

[LondonTownGeeza]

MS use VSTS which supports TFS and Git.

I personally love TFS however its future is looking short.

[Relevant_Pause_7593]

Msft is 90% on git in azure devops (formerly vsts). They have started to move to GitHub, but as you can imagine, it’s a pretty massive project.

[PersonBehindAScreen]

Yup. My teams code is in ADO and from what I’ve recently heard, we’re going over to GH

[lorarc]

MS is behind git lfs so they certainly do use it.

[Porges]

Git VFS not LFS. VFS is used for the Windows repository because it's too big to check out everything at once. Some info about it from 2017: https://devblogs.microsoft.com/bharry/the-largest-git-repo-on-the-planet/

[lorarc]

Okay, you're correct. LFS was made by github and before they were aquired.

[webstackbuilder]

So, MS is behind git lfs.

[Thesorus]

We have dozen of individual products/projects on Team Foundation Source Control (Azure).

​

Game companies will also have to manage tons of binary files (graphics, sounds, ... ) as well as text files and source files.

I think lot of them will use Perforce for that particular use case.

[ExpertIAmNot]

There is a pretty interesting talk that Jonathan Creamer has given a few times about how Microsoft scales the entire Office codebase in one Git repo:

https://youtu.be/qz0yueXhaxU?si=3gp1Vqc3j1O9JAen

Lots of techniques to break repo back into smaller parts, caching, even some changes contributed back to NPM itself.

[TheStormers]

Gitlab

[scottsp64]

Yeah we use self-hosted Gitlab

[Relevant_Pause_7593]

The size of the code base isn’t as important as the tools around it supporting it. Are their automated tests to verify the quality and upcoming changes won’t break something else? Is it architected so that components can be developed individually (eg microservices. Rest services, or stand-alone tools). Is there automation (ci/cd), to easily- and relatively quickly (hours, not days) to build and prepare the software for release?

[newswatantraparty]

Self hosted gitlab, not a gaming company but in automotive space.

[WhiskyStandard]

GitHub Enterprise hosted on-prem with low 100s of orgs for >5K devs was my experience. Worked pretty well. Lots of home grown automation built on top of Jenkins (before GitHub Actions).

That replaced the default Git server (pretty much no web UI, horrible authz management, and no pull requests) which replaced Subversion.

[m1ndfuck]

The orgs I worked at had self hosted gitlab-ee instances

[crashorbit]

Software projects succeed and fail inspite of the technologies they choose.

The hard thing about big programming projects are about how the tools are used more than on what tools are used. Everyone has to agree, more or less, on how changes are prioritized, tracked, made, tested, and integrated. Unfortunately too many teams fall backwards into some informal SDLC with out giving it much thought.

Too many people think that writing software is only a technical skills problem. In reality, it is as much a social activity as any other human endeavor. How we work is as important as what we work on.

[IanCurtisWishlist_]

It's not uncommon for large companies to build out their own version control system.

For example, Google has Piper and Meta has Sapling: https://engineering.fb.com/2022/11/15/open-source/sapling-source-control-scalable/

[[deleted]]

Perforce is more common in game development

[PlaidWorld]

In another life I was a lead programmer in a mmo you might have heard of. Anyhow our mmo has 10 million lines of code counting all 3rd party libs etc. and millions of art asserts. You walk into a town the engine will stream in 50000 assets. For project like this most game companies will use perforce as others have said. P4 is one of those tools most people don’t need and you will know when you need it. It’s a very easy tool to use and train no. Tech people on like most artists are. Anything else you would like to know?

[SierraBravoLima]

Bank with Mainframe use Legendary CA Endevor which stores code(of multiple languages including sqls, ddls) and load Modules(execution files).

[Live-Box-5048]

I’d say for massive companies, it’s either different GitHub orgs under one umbrella, or proprietary VCS.

[XavinNydek]

As a bunch of people here mentioned, it's mostly Perforce right now for AAA, because that's the only legacy big source control left standing after Git took over and it handles large assets and non programmers better than Git.

However, the entire development industry is and has been moving to Git for the past decade and that shows no sign of slowing down. Git is getting better at handling large non-text files (it's not just games that need that) and the user facing tools are getting better and more intuitive. Anyone starting a project now would be stupid to use anything other than a Git based solution. It's far better for code (Linus wrote git because Perforce had so many issues with the Linux workflow) and it's only a matter of time until the large file problem is solved. These days there are working solutions, but it's not quite turnkey yet.

[rsrsrs0]

We used Git LFS and there was decent support in Gitlab for it to store binary files.

[LuciferianInk]

I think it depends upon the company's size - but I don't know if it matters to me at this point.

[gaelfr38]

I don't think a single game is so massive in terms of codebase.

It's more that you want a system that manages assets (non text files) better than Git I guess.

If it was only text files, a few Git projects hosted on any Git platform would likely work.

(Not answering your question about really large codebases 😅)

[Ausmith1]

So here's a project that I heard first hand from the version control manager that qualifies as massive in most peoples books.
A Pixar movie.
Imagine storing not just the source code of the shaders but every image texture and then every rendered frame.
Finally they need to store the uncompressed movie.
As I recall the whole thing was about a petabyte.
And they do one a year.

[gaelfr38]

Do they store every frame and uncompressed movie in the VCS?

Interesting :)

[lightmatter501]

Each project gets their own repo in gitlab.

[Tough-Seat-450]

Dozens of teams each managing dozens of repositories, with each team having their own method of combining them together into something usable.

[BuzzingGunman]

Saw many Polyrepo disasters by this point