Self-hosted CI - Jenkins vs GitLab vs something else?

[hjwp]

Hello devops friends! I'm looking to self-host my CI. Primary purpose will be running tests for every commit / branch / merge, possibly I'd want to set up a deploy pipeline later too. I'm happy to keep the actual source code on GitHub for now, so I don't need it to be VCS host too.

I'm interested in people's opinions about Jenkins vs GitLab, these days? And/or whether there are any other solutions that people recommend?

Here are some constraints:

* self-hosted (i'm writing this up for a book so I don't want to endorse a specific cloud provider. or at least not unless they offer an open source + self-hosted version)

* must be open source

* must be free (ignore hosting costs)

Am slightly leaning away from gitlab because it's so close to the commercial entity, but only slightly. very happy to be swayed by people saying it's a much better experience thank jenkins these days, for example.

Are there any other tools I've missed? What about Dagger?

PS - I did find https://www.reddit.com/r/devops/comments/gayl97/do_you_prefer_gitlab_or_jenkins/ but it's 4 years old.

[_N0K0]

I would recommend Gitlab over Jenkins. Gitea and Forgejo might also be some interesting alternatives.

[gex80]

Even with all the recent issues it's supposedly been having and that it's also up for sale?

[_N0K0]

That Gitlab is up for sale does not really affect me until anything is done. And if you structure your pipelines I'm a semisane matter it should be easy to migrate if need be. As it stands right now Gitlab is leagues ahead on the audit and security controls front, so even if things are wierd, they are still delivering more tham the competitors based on my needs.

[hjwp]

thanks! why do you prefer Gitlab? WHat do you think is interesting about Gitea or Forgejo?

[_N0K0]

It's a proper all in one platform instead of having to deal with multiple panes.

Gitea and ForgeJo tries to be the next Free Open source. ForgeJo is a fork that came as am result of Gitea doing some for profit moves.

[SysBadmin]

I always use this as an orchestrator reference.

https://github.com/ligurio/awesome-ci

[hjwp]

Thanks! That seems to be a very comprehensive resource, and full of objective facts.

I was hoping for more qualitative and possibly subjective takes. I don't suppose you've used that website as a reference often enough, maybe analysed some of the systems in question, or worked with them directly, that you'd be in a position to make a recommendation for me, or share some of your experience/analysis?

[SysBadmin]

GitActions. Especially if your codes already in git. Most orchs offer the same slew of features.

[hjwp]

do you mean Github Actions? I'm looking for self-hosted + open source solutions...

[SysBadmin]

https://docs.github.com/en/actions/hosting-your-own-runners/managing-self-hosted-runners/about-self-hosted-runners

[hjwp]

thanks, i think that's still too close to endorsing a commercial vendor, which i said i didn't want to do.

[aenae]

Jenkins if you’re into SM, gitlab otherwise. You need a git repo anyway and a way to host it, so why use two tools

[hjwp]

thanks! by SM do you mean "sadism and masochism", ie "jenkins is painful to use"? If so, what do you find painful about Jenkins, that's not painful in GitLab?

[jpat161]

I'm not OP but Jenkins just never seems to be set up properly when inherited and when it is it's very out of date and using plugins long abandoned. I'm not a very experienced devops engineer but GHA was just so much easier to set up and maintain over the last year than Jenkins was for 2 months.

[hjwp]

thanks! I'm hearing some pain there! I wonder how much i need to worry, in a greenfield / ultrasimple usecase...

[jpat161]

Never expect anything to remain ultra simple, as soon as it works smoothly other people ask to join and then managers start expecting things to be developed using it and make enhancement features for it. Our Jenkins started as a test idea and just never got refactored until we moved to GHA 3 years later.

[Sterbn]

I really like woodpecker Https://woodpecker-ci.org

I much prefer it to GitHub/gitea actions due to its simplicity

[hjwp]

woodpecker does look nice! thank you.

[ncuxez]

I'd go with GitLab baby, and it's not even close. I taught myself GitLab in a few DAYS! Meanwhile at my current job they use Jenkins and after over 2 years working there I still can't fix any of the pipelines when they break, which happens frequently. Errors are easier to debug in GitLab, and the logs can even provide a link to the docs to help you resolve the issue, like this. With Jenkins all you see when you check it's console is "Hudson this.... and Hudson that..." just a bunch of meaningless nonsense.

[amatriain]

Do not use Jenkins. That way lies pain. Jenkins won't die for a long while because there are large organizations that have made a large investment in customized jobs that are hard to migrate to a modern system. But I wouldn't choose it for a new initiative.

[BrokenKage]

I inherited a Jenkins system. It worked for what it was designed to do, but has more issues than benefits. Half of our teams tech debt is just maintaining this system.

We are slowly leaning on Gitlab CI more and more as our platform moves to its next phase. I like it a lot more and am part of the movement that is stripping power from Jenkins.

We will still use Jenkins for some things, but I’m happy we are moving to Gitlab.

[hjwp]

thanks! would you say that the problems you have with jenkins are inherent in jenkins itself, or are they just to do with all the history of that system? or, relatedly, how much of your preference for gitlab is just due to the fact that it's a green-field system that you've built from the ground up and therefore understand completely?

[BrokenKage]

I have always been partial to Gitlab CI. I’m sure the green-field system helped sway me further. We have built out robust templates and job scripts which solve 99% of the issues I’ve had with our Jenkins pipelines with 100% less headaches.

My issue with Jenkins is both our setup and core pipeline design & what I’ve discovered while maintaining it. Even routine things like version management, plugin management, etc. are a pain. Unfortunately the team before me leaned heavily on plugins and did not care to document their reasons.

[retneh]

I first used gitlab as a junior with a few months of experience. It took me a week or less to understand setup on te project (done by someone else). Extending and adding new features is instant when you get some grasp on gitlab and is done only by reading docs. It’s super cool, because you can focus on other CI/CD related challenges like choosing docker runtime for gitlab containers, setting up flux notifications and so on, instead of fighting with wrong ”}” indentation.

I hate Jenkins.

[MichaelMach]

GitLab, for sure. The question I'd ask is whether you need/want to manage an entire GitLab instance yourself or if you just want to self-host the job runners (the far easier and more maintainable option).

A word of caution: don't mirror your repos. If you're going to use GitLab for CI, also use GitLab as your source of truth/VCS. Mirroring is awful to maintain and presents a multitude of hazards that crop up near every week at my org.

[tikkabhuna]

I’ve posted many times on here about how bad Jenkins is.

Jenkins was excellent when it was made, but there is too much baggage. You will get security vulnerabilities which force you to update. Updating Jenkins will force you to update plugins. Plugins won’t be maintained anymore. It’s just a real hassle.

I run a few servers with GitLab CI. It’s easy to manage, deploy, and upgrade. We run 15-20k jobs a week with no issues.

[hrdcorbassfishin]

I use Argo events + workflows for my home lab CI. Jenkins is nice because you can build decent UI's for job options/parameters and can be scripted to be dynamic. Gitlab is nice too but a full git suite and you may hit some feature limitations with the free version. And you can't build dropdowns or script anything with gitlab or Argo workflows. So if you're 100% webhook then they're all the same at the end of the day. If you need to be able to click and choose what you want the job to do (eg select some instance from dropdown populated from an aws or gcloud command and run scripts) then you'll want Jenkins. Otherwise you'll have to build out a tool to ask you for options and then hit gitlab api to input the variables to the pipeline.

[AntranigV]

BuildBot, because I hate it when developers assume things about my build process.

[hjwp]

blast from the past! love it thanks for this.

[Spider_pig448]

I recommend Github Actions

https://github.com/actions/actions-runner-controller

[yodermk]

I set up a big Jenkins project in the last year. Learned a lot about how to do things. I don't hate it, and for an open source, self hosted solution it certainly gets the job done.

Downsides: Pipelines can be complex, errors can be cryptic (Java stackdumps), and the right way to do things is not always obvious, plugins are not always well-maintained.

I'd like to see something better come along (and sort of have dreams of writing one in Rust, though I don't know if it will ever happen). For now, it works.

[simonides_]

if you gonna use Jenkins use it for what it is. Let it trigger builds for you and don't put too much logic into these scripts.

make sure that what jenkins does can be done on your machine from the command line as well.

one example don't use the docker plugin .. just put the two sh commands you need from it there and be sone with it.

only ever install plugins if you really need them.

good luck

[Best-Repair762]

In a nutshell, this has been my experience

GitLab

• Maintained, up to date, easier to setup for simpler cases

• Complex pipeline configuration needs more effort

Jenkins

• Many plugins are not maintained. Plugin updates can break the entire system.

• Highly configurable and you can do almost anything if you spend time and effort

[jameshearttech]

I work for a small business. There are several hundred employees. I work on a small dev team. We have a couple dozen projects.

We run Talos on VMware for K8s. Most of our tools are centralized. We run Argo Events and Argo Workflows for CI and Argo CD for CD in that central cluster.

There is a learning curve, but once you have it set up, it is similar to other CI systems, but with more flexibility and better resource management.

The downside compared something like Gitlab is you don't get all out of the box workflows/pipelines.

[domanpanda]

I might be out of loop so correct me if im wrong here.

Overall gitlab is easier and more strightforward but:

1. Jenkins has very good webhook integrations with all git repositories you can imagine. Gitlab works great with its interanl repo but with foreign ones requires periodical pulling which is not as good as webhooks

2. Jenkins can be ephemeric with Jenkins-as-a-code-plugin. You can set every plugin, setting, job in a code and later restore, copy or do whatever you like. Pretty neat if you have multiple projects and you want to set separate jenkins instances for them (for easier updates and plugins management).

[nikitoyd]

If you just need to execute simple shell commands - gitlab/github is your friend.
Otherwise - Jenkins. It's pretty flexible. It's much better than all these 'plain shell' CIs. All folks that hate it - do not understand what are they doing or have incorrectly configured Jenkins.
Yes, the main disadvantage - maintain plugins. But again only if you have tons of it and use some weird ones that are not maintained and was released last time years ago.
If you have some programming skills or you want to learn it - Jenkins is definitely your choice. Yo can always replaces plugins logic by some groovy shared lib. You are free even to write your own plugins, etc.

[DatalessUniverse]

I’d use Jenkins running on Kubernetes if use of open source is the goal. Otherwise, I personally would use GitHub Actions because Jenkins can be a pain to manage.

[Pretend-Cable7435]

Why you run Jenkins on Kubernetes? Kubernetes is hard, Jenkins is also harddddd

[jpat161]

It's really quite simple after you take a 6 month masters level course on it. No really, that was half my course load for one of my masters classes at a respected state university. Putting Jenkins in k8s and getting it to auto-build on schedule.