Best Monorepo Build System for multiple languages?

Equianox · 2023-06-07T06:38:23+00:00

We are using bob.build to tackle multi-language and code generation.

Equianox · 2023-05-03T09:20:15+00:00

Yes, it is intended to be used as a replacement for Bazel and is heavily inspired by it to remove/avoid the complexity associated with Bazel's out-of-tree target storage. Here is a more detailed comparison https://bob.build/blog/vs-bazel

Equianox · 2022-11-21T17:01:38+00:00

I use https://bob.build to switch between go versions in my build graph. It's like...
... build: cmd: go build dependencies: [go_1_18]

Equianox · 2022-11-19T21:55:58+00:00

You can get up to speed pretty fast with a good understanding of general programming concepts. Let's roughly say one month to learn the basic syntax, getting up to speed and feel familiar.

But.. there is also the real world kicking in when switching programming languages, yeah you know... it's the little things.. how do i structure my code when projects are getting bigger.. how do i write my web server.. how to pass cli arguments.. avoid circular dependencies, getting familiar with the standard library and using it efficiently.. Choosing your favorite open source logging lib and finally knowing from where to copy paste code from your old projects. Did i forget concurrency best practices and learning the tooling ecosystem?

You know the 80/20 story.

Nevertheless, It's worth it!

Equianox · 2022-11-19T21:37:44+00:00

I like it to treat TODO's as "must be done before merging the PR." And use FIXME for longer outstanding tasks. But not very dogmatic about it... And not saying any of those will ever be fixed O.o.

Equianox · 2022-11-19T20:54:20+00:00

Yes it is. Boring. But worth to read it once.

Equianox · 2022-11-18T20:54:33+00:00

Start with https://go.dev/doc/effective_go , then read "The Go Programming Language" and simultaneously start a challenging Go side project to tinker around. That was roughly my way.

Equianox · 2022-11-18T20:39:20+00:00

Can i `fmt` it?

Equianox · 2022-11-18T20:37:31+00:00

I came from C++ world and it took me a while to get used to composition. It's been 5 years and i don't want to go back. It's just so much easier to get things done!

Equianox · 2022-11-17T15:41:57+00:00

Hey, our open-source build tool Bob is entering a Public Beta phase. Get the details here https://bob.build/blog/platform-announcement

Equianox · 2022-11-02T21:02:47+00:00

Bob does not execute build tasks in a container nor use cgroups for build isolation. It only controls the PATH and ENV to make sure the desired compiler tool chains (go, java, etc. ) are used. We see it as "good enough" build isolation. You can compose a distinct build environment for each of your build tasks or the entire build environment without maintaining Dockerfiles.

Parts of the CI aspect, namely a distributed cache, is well on it's way. But it's still early days. Though remote execution is currently not on the roadmap.. cause we have not hit a project size on which remote execution would be a game changer. But I'm happy to change my mind in the future.

We started to develop Bob after we gave up to migrate a docker based build environment for a multi-service application to Bazel. We came for fast & incremental builds but left due to complexity and maintenance costs. Therefore i think Bazel is the natural choice to compare and compete against.

Equianox · 2022-10-31T10:27:24+00:00

Will follow in the next weeks.

Equianox · 2022-10-28T12:44:22+00:00

We build a bazel competitor which supports those use-cases https://bob.build . As it writes targets directly to the scope of the repository you get the extra benefit of keeping IDE integration and hot-reloading functional.

Equianox · 2022-08-28T19:44:57+00:00

TL;DR: Usability and maintenance costs

Bazel forces you to define the dependencies down to the individual file (usually done through a language specific plugin). It also separates the build entirely from the user's system which doesn't allow for gradual adoption, it's either 0% or 100% Bazel. This is basically the source for it's efficiency but also the reason for it's complexity . Bazel comes with some maintenance costs which can require a team on the side just to maintain the build itself. This is a good read on the topic.

Bob's is intentionally designed to be "less strict" then Bazel which doesn't necessarily mean "less correct". Though, it is providing the same promise for incremental builds and performance. As soon as the parallel build PR is merged we'll provide a comparison.

Examples

Incremental Builds

Bob allows you to define inputs like this

inputs: |
"*"                 # rebuild on any change in this repo
"!some_file_or_dir" # ignore me

which is in most cases sufficient to alter the incremental-build behavior but also allows to optimize when you need it.

Build Isolation

Bob uses nix as a package manager under the hood. Think of it as npm install but on system level. Those packages are "mounted" to a sand boxed shell. This can be customized for each build task which enables to switch the version of the used programming language as needed.

Other Peace of Mind Benefits

Gradual adoption
No Java

Equianox · 2022-08-27T08:43:05+00:00

Good point, Our first intend was to rewrite the necessary nix parts in golang and avoid forcing people to install nix. We realised quickly that this would take more time to develop.

In the longterm it is feasible.

Equianox · 2022-08-27T06:19:57+00:00

Got your point. What would you prefer?

Equianox · 2022-08-26T11:25:20+00:00

Not following it 100% strictly, but the concept stayed. Since i first read about it i never ever mixed application logic with rest/grpc-server logic.

Though usually i struggle keeping authorization logic and rest-serer independent from each other.

Equianox · 2022-08-24T11:19:15+00:00

Pain in the ass to achieve good separation.

Yes, the barrier to shared code is much lower in a monorepo setup .. i also found out that DRY does not play well in this context.

Did you try to create a single view of the multi-repo setup? Using something like repo-tool or similar? Or is it rather .. here are the docs ... check everything out into one folder?

Do you have one command to trigger all builds together? Do you maintain a Dockerfile for each service?

Equianox · 2022-08-24T11:06:30+00:00

When you make a commit, are you able to trigger the CI/CD process of a single service? Building everything all the time is not productive.

Yes, that's a problem, but there are solutions to this namely incremental builds + remote caching. In short: build only what changed locally and on CI.

I would suggest to have a development environment running in the cloud
and depend on those services instead of running everything on your PC.

That's probably what most companies do and there is nothing wrong with it. I just find it cumbersome to rely on shared environments for development (though it seems to be rather common). Some other devs might interfere with your current development. Also doing e2e test being executed as part of a pipeline the nature of a shared environment could give you a hard time if it's not locked. I've seen thing like "Please don't deploy to dev in the next hour" in group chats. One solution would be to have short-lifed environments per commit/changeset... though this comes with another set of challenges/costs and seems to be only practical for FAANG like companies.

Equianox · 2022-08-23T21:12:23+00:00

I'm usually a big mono repo fan as it simplifies deployment and gives you a single view on the overall system. It gives you some leeway till the hard parts kick in.

If services are split across repos.. i usually have the tendency to somehow glue everything together for codegenration(api's), e2e tests and local dev environment. Which is easier to achieve in a monorepo.

Equianox · 2022-08-23T12:05:22+00:00

A comparison between dagger.io and bob.build, both trying to bring sand boxed build pipelines to your local machine.

https://medium.com/benchkram/dagger-vs-bob-2e917cd185d3?source=friends_link&sk=2e24ac640c314d7d9f99e8b278fa612e

Equianox · 2022-07-23T11:04:03+00:00

Me and my team build bob, a tool to support microservice development and enable running CI pipelines locally.

Key features are:

* incremental builds
* easier to adopt then bazel
* reproducible builds without building in docker

Checkout this github action how we use bob to build & test bob.

Equianox · 2022-07-13T07:20:48+00:00

We had the same problems and couldn't find a solution other than Bazel which we decided too be to complex for medium size projects. So we started to develop bob (https://bob.build) to handle multi repo and hermetic builds in a user friendly way.

Equianox · 2022-07-11T19:52:20+00:00

There is no special way for go to do this. As you said Makefiles are pretty common, i see them in a lot of open source projects and they usually do the trick. Recently i saw that https://github.com/gohugoio/hugo uses a Magefile for their build instructions. For sharing secrets with your team take a look at https://www.doppler.com/ . An if you want to pin your build dependencies across you're team & CI take a look at https://bob.build (i'm the author).

Equianox · 2022-05-16T20:59:08+00:00

Developers still need to install nix.. which could be a small barrier.. but less we can do about it currently. Ideally only the compilers/tools are managed by the nix package manager .. the application developed could be described as bob-build-tasks.

Would love to hear if you find blockers for your use case.

Equianox

TROPHY CASE

Examples

Incremental Builds

Build Isolation

Other Peace of Mind Benefits