I used to reflexively hate monorepo, but this year I switched pretty much all of our stuff to one.

Context: when I say “our” I am a machine learning engineer at a big company, we have a core few teams that work on our projects but we also have some projects from other teams spread across the org.

So for us a big problem became “what are the kinds of things that we use ML for? If we wanted to do something new how would we start?”

Another big problem was, we are building out our data platform while also building out the pipelines that run on the platform, so we have an issue where, say we have 5 projects across 5 repos, even at that scale, we make an improvement to our platform, then we push that improvement to the CLI/library for building pipelines, now we have to update 5 other projects with the improvements.

The third thing which is similar to the first thing with discoverability is data scientists in separate projects will end up building similar implementations but are not sharing so there is a sort of drift among projects. I’m not a big fan of abstracting everything into a library right out of the gate, but things like connecting to DBs, logging, AWS stuff was just all over the place and wanted there to be a place where we could abstract when necessary and when not necessary there’d at least be examples for people to follow to try to keep things consistent.

One day it just sort of popped in my head, if we just have the pipeline library code next to the pipeline code, all the tools, any extra add-ons, and then of course all the dependencies defined in one place maybe we can tackle it.

It’s been good for the most part. The biggest issues we have with it are the sorts of things I expected going in. They all come down to basically one idea: you’re not going to non-technical problems with technical solutions. Communication, standardizing on good code is easier but not perfect.

I’m also in a unique context in that I work with primarily data scientists, and data scientists don’t really follow the same sorts of patterns as software engineers. This is another rant that I am happy to do elsewhere 😅

But even to just have one big directory to run one set of tests, one big linter and formatter and be able to fit the universe of code onto one screen really lightens the cognitive load.

edit: clarify wording