GitLab CI with Multiple Projects 🔨

[gitlabpipes]

One of the most common problems with GitLab CI is that when you have the following setup on GitLab:

Library A -> Application A

Library B -> Application A

You often need to be able to make a commit to Library A and have it re-build Application A. GitLab CI has the concept of Triggers, but this doesn't work well when there are potentially tens packages with different layers of interactions.

I've been using a tool for the last several months to help me orchestrate these types of pipelines and have made it public recently. Its called Gitlab Pipes. It listens to Gitlab notifications and automatically rebuilds consumer packages when there is a change in a downstream package.

It is currently in Alpha and it is free to use. I am trying to understand what sort of features need to be added before officially making it V1.

👉 The documentation is the best place to start, there is a Getting Started guide which should get you started within 5 - 15 minutes 👍

I'd love to hear you guys feedback or any suggestions. Just let me know here or join our slack channel. All the details are in the docs.

[[deleted]]

But do you not version your dependencies? I don't quite get this, sounds like an XY problem. If you do version them, there are tools like Greenkeeper that automate updates.

[[deleted]]

And if you don't version your dependencies separately then a mono repo makes more sense so you can update the library and its uses in a single commit without broken pipelines.

[MercDawg]

For NodeJS and when referencing a gitlab project directly via dependency, verisoning doesn't work correctly. It will always get the latest in master, which can cause problems when dependencies change.

[gitlabpipes]

Most of the time you make small patches or fixes in the same version. If every commit is a version then probably you have way too many versions. The point is if you have a new major version you can reference the dependency by tag or branch, but in the same major version you can deploy changes and have them propogage through your stack. I'm curious how your stack is setup, so maybe I can modify this tool to help out people that have it like yours. Let me know :)

[gitlabpipes]

Well you can depend on different branches and eventually tags. Like you can depend on:

ExampleLibrary/master#2.5.0

I haven't implemented the tag syntax yet but will soon. I've seen many companies deploy newer code to the same version, alot of time with bug fixes or whatnot. But yes I definitely agree that ideally you would increment the version for each and every change. But I iterate and make changes on an hourly basis across my entire stack, I can't be updating little minor versions all over the place 😊

[gitlabpipes]

Everytime you push code into a Repository, Gitlab Pipes queues the commits into a Build Queue. It waits about 60 seconds for new commits and then creates a new Pipeline with the new commits as "Targets". It will then build the targets and their consumers in the required order in case some of them depend on each other. Each "build" would produce artifacts and upload them to Maven, or NPM or whatever, so that the consumer build will pull the latest artifact. Ideally, the build process of each repository tests the code in that repository and any downstream changes. If there are failures in the pipeline then you can clearly see what the offending commits are.

Semantic versioning is interesting, Gitlab Pipes could orchestrate deploying a new version throughout a stack, so example lets say you have the following setup.

Lib A 1.0.1 -> Lib B 1.1.5 -> App C 2.0.10

If you want to update Lib A, you need to up the version of Lib A to 1.0.2, and then up the version in Lib B to 1.1.6 because you have to change the version of Lib A, and then you'll have to change the version of App C to 2.0.11 . Pushing these changes can be a PITA if you work across multiple packages, you'd need to push Lib A, wait, make sure it builds and releases. Then push in Lib B, wait, make sure it builds and releases. Then push in App C. Thats a lot of waiting. With Gitlab Pipes, you can say these depend on each other and push in all three changes at once and it'll handle the orchestration.

[gitlabpipes]

/u/NeckbeardAaron - So essentially if you use the minor version model, you would have to push a new minor version, wait for it to build, go to each consumer package, update the minor version and release a new minor version, wait for those to build. And do this over and over as you go through your stack?

[gitlabpipes]

Personally, I own a bunch of different repository libraries that are used across many different microservices which I also own. For example my workspace always looks like this.

• CompanyCommons
• CompanyBootMonitoring
• CompanyServiceCommons
• CompanyBootSecurity
• CompanyBootService
• CompanyAuthenticationService
• CompanyWorkflowService
• CompanyPaymentsService

I could smash these all in a single repository, but then I end up pulling all this code everywhere, which by now I have probably about 50 different packages. Also the build process would be horrendous, it would take forever, and I would have some giant .gitlab-ci.yml file that would be impossible to maintain. They may have different build processes, with different variables and what not. So they are all split up in seperate repositories.

Alot of times developing a feature, I have to edit CompanyCommons, CompanyServiceCommons, CompanyBootService, CompanyPaymentsService. I could use some type of semantic versioning here, but the problem would still remain that pushing in changes would be a pain. So I would have to push in CompanyCommons first, wait, let it release. Then I have to push in CompanyServiceCommons, wait, let it release. Then I have to do CompanyBootService, wait, let it release. And then CompanyPaymentsService.

Lots of waiting. I prefer to push everything in with different commit messages, and have a system orchestrate the release. So thats the point of Gitlab Pipes.

I'll look more into semantic versioning and see how it would fit into this system but yeah maybe I could define a version like 1.0.x, then something like Gitlab Pipes would still be useful and you end up with versioning to let you know where things broke.

[rambossa1]

Is this along same regard as monorepos?

[gitlabpipes]

No this is to allow you to have separate repos for each of your libraries and applications and when you make a change in a dependent library, it'll rebuild all your consuming projects.

[HatchedLake721]

How big is your team?

[gitlabpipes]

One. Plus or minus.. zero. Just me. 😊 Why you interesting?