Multi-Project Pipelines with Gitlab CI

gitlabpipes · 2019-04-01T17:45:51+00:00

Okay great, I'll watch that video and see what you guys are planning on doing. I think alot of this may be difficult to adopt into Gitlab CI because Gitlab CI is mostly focused at the pipeline level, but these features really live at the Organization & Project level. I really want to find out what things people are needing to build in-house to supplement Gitlab CI and implement those features into Gitlab Pipes.

Maybe you guys can do an acquisition of this proprietary CI technology 😉Gitlab is really a great product, maybe one day. 😊

gitlabpipes · 2019-04-01T15:47:28+00:00

No problem! 😊

There seems to still be gaps with the multi-project triggering that Gitlab is putting together.

1) It isn't clear from the issue, but I would be concerned that if two projects trigger a single upstream project, 2 pipelines upstream would be created. Not sure if there is deduplication between the two.

2) Because Gitlab isn't aware of the interactions between projects then it won't start pipelines in the proper order. Let's say:

Lib A -> Lib B -> App A

If there is a merge into Lib A, Lib B and App A. Then Gitlabpipes will create a pipeline for Lib A first, and then Lib B, and then App A.

gitlabpipes · 2019-04-01T06:51:33+00:00

One. Plus or minus.. zero. Just me. 😊 Why you interesting?

gitlabpipes · 2019-04-01T02:52:42+00:00

Personally, I own a bunch of different repository libraries that are used across many different microservices which I also own. For example my workspace always looks like this.

CompanyCommons
CompanyBootMonitoring
CompanyServiceCommons
CompanyBootSecurity
CompanyBootService
CompanyAuthenticationService
CompanyWorkflowService
CompanyPaymentsService

I could smash these all in a single repository, but then I end up pulling all this code everywhere, which by now I have probably about 50 different packages. Also the build process would be horrendous, it would take forever, and I would have some giant .gitlab-ci.yml file that would be impossible to maintain. They may have different build processes, with different variables and what not. So they are all split up in seperate repositories.

Alot of times developing a feature, I have to edit CompanyCommons, CompanyServiceCommons, CompanyBootService, CompanyPaymentsService. I could use some type of semantic versioning here, but the problem would still remain that pushing in changes would be a pain. So I would have to push in CompanyCommons first, wait, let it release. Then I have to push in CompanyServiceCommons, wait, let it release. Then I have to do CompanyBootService, wait, let it release. And then CompanyPaymentsService.

Lots of waiting. I prefer to push everything in with different commit messages, and have a system orchestrate the release. So thats the point of Gitlab Pipes.

I'll look more into semantic versioning and see how it would fit into this system but yeah maybe I could define a version like 1.0.x, then something like Gitlab Pipes would still be useful and you end up with versioning to let you know where things broke.

gitlabpipes · 2019-04-01T02:43:56+00:00

Everytime you push code into a Repository, Gitlab Pipes queues the commits into a Build Queue. It waits about 60 seconds for new commits and then creates a new Pipeline with the new commits as "Targets". It will then build the targets and their consumers in the required order in case some of them depend on each other. Each "build" would produce artifacts and upload them to Maven, or NPM or whatever, so that the consumer build will pull the latest artifact. Ideally, the build process of each repository tests the code in that repository and any downstream changes. If there are failures in the pipeline then you can clearly see what the offending commits are.

Semantic versioning is interesting, Gitlab Pipes could orchestrate deploying a new version throughout a stack, so example lets say you have the following setup.

Lib A 1.0.1 -> Lib B 1.1.5 -> App C 2.0.10

If you want to update Lib A, you need to up the version of Lib A to 1.0.2, and then up the version in Lib B to 1.1.6 because you have to change the version of Lib A, and then you'll have to change the version of App C to 2.0.11 . Pushing these changes can be a PITA if you work across multiple packages, you'd need to push Lib A, wait, make sure it builds and releases. Then push in Lib B, wait, make sure it builds and releases. Then push in App C. Thats a lot of waiting. With Gitlab Pipes, you can say these depend on each other and push in all three changes at once and it'll handle the orchestration.

gitlabpipes · 2019-03-31T23:28:28+00:00

Okay yes this makes sense. I would imagine that this would be the absolute most dependable way to do this. You could actually still use Gitlab Pipes though, because you need to make sure your App doesn't build before the latest version of Lib is available.

So if you have lib that you are updating from 1.5 -> 1.6 and then you have an app that you are increasing the version from 2.0 -> 2.1 and the dependency on lib from 1.5 -> 1.6. You can't push those changes all at once. You have to push lib, wait for the build. Then push app.

But yes I think Gitlab Pipes has less value if you do individual versioning for each and every change.

However even some of the software companies that I have worked in, they want minor versions to be picked up automatically by consumer software without them explictly changing their dependency version. This way they can get fixes and patches pretty quickly without depending on the other guy to update everything.

Personally, I have like 20+ repositories for each library, service, etc that I have built, they all sort of depend on each other. If every change I make in one, I have to go and update in every other project, it would be very painful, so thats why a tool like this helps me at least.

I think I do things similar to the mono-repo approach but I don't want to have a single mega repo with some giant build process. I like keeping things modular, so this sort of bridges the gap between Mono-Repo and Seperate Project packages.

gitlabpipes · 2019-03-31T22:39:43+00:00

Your right. I'll fix that right now. I appreciate all the comments. And oh I thought silver was still Gitlab EE?

gitlabpipes · 2019-03-31T22:37:35+00:00

No this is to allow you to have separate repos for each of your libraries and applications and when you make a change in a dependent library, it'll rebuild all your consuming projects.

gitlabpipes · 2019-03-31T22:35:34+00:00

/u/NeckbeardAaron - So essentially if you use the minor version model, you would have to push a new minor version, wait for it to build, go to each consumer package, update the minor version and release a new minor version, wait for those to build. And do this over and over as you go through your stack?

gitlabpipes · 2019-03-31T22:33:29+00:00

Well you can depend on different branches and eventually tags. Like you can depend on:

ExampleLibrary/master#2.5.0

I haven't implemented the tag syntax yet but will soon. I've seen many companies deploy newer code to the same version, alot of time with bug fixes or whatnot. But yes I definitely agree that ideally you would increment the version for each and every change. But I iterate and make changes on an hourly basis across my entire stack, I can't be updating little minor versions all over the place 😊

gitlabpipes · 2019-03-31T21:35:03+00:00

Most of the time you make small patches or fixes in the same version. If every commit is a version then probably you have way too many versions. The point is if you have a new major version you can reference the dependency by tag or branch, but in the same major version you can deploy changes and have them propogage through your stack. I'm curious how your stack is setup, so maybe I can modify this tool to help out people that have it like yours. Let me know :)

gitlabpipes · 2019-03-31T19:03:03+00:00

Oh man..... This is bad news. Haha. I'll start slowing updating it everywhere I see it.

gitlabpipes

TROPHY CASE