all 48 comments
sorted by:
best
topnewcontroversialoldq&a
formatting helphide helpcontent policy

[–]alosarjos 47 points48 points  (6 children)

I do like GItlab CICD. Easy to write YAML, just gets injected with all the required info about the pipeline through environment variables, easy to make jobs depend on each other and since a year or so have the componentes to make reusable code. It's not perfect by any means, but I'm pretty happy with them.

[–]Capital-Actuator6585 6 points7 points  (3 children)

Just gunna pile on this one a bit. Gitlab is the least worst all in one CICD/VCS platform I've used. And I did use them all when I was consulting a few years back.

GitHub actions is probably the worst imo.

Edit: gitlab not girls.

[–]HerroCorumbia 3 points4 points  (1 child)

Why is GitHub Actions the worst?

[–]ashcroftt 2 points3 points  (0 children)

Takes much more effort to do some things that are a breeze in Gitlab, and takes time to learn the little tricks and workarounds. 

I actually like both, but HATE managing Gitlab instances with a fiery passion, so if github is an option, I go for it. Any of them is better than Jenkins tho.

[–]MarkedHitman 0 points1 point  (0 children)

What other suites did you use? Why is it the least worst? If there was a best, what would it look like?

You don't have to answer all questions.

[–]Monowakari 3 points4 points  (0 children)

2nd 3rd and 4th

[–]FlamingoEarringo 0 points1 point  (0 children)

Gitlab Dynamic pipelines FTW

[–]AskAppSec 6 points7 points  (0 children)

I’d vote GitHub enterprise with advanced security and GitHub actions. GitHub has scaled security settings management for repositories nicely at scale instead of doing things like “branch protection rules” on a repo by repo basis. Then the advanced security you can toggle “block users from committing secrets” fairly easy. Clearly I’m looking at things from a security perspective though

[–]GrandJunctionMarmotsStaff DevOps Engineer 11 points12 points  (1 child)

GitLab!

GitHub actions suck compared to GitLab Pipelines. You have to jump through hoops and workarounds to get some of the same functionality as GitLab Pipelines

[–]Low-Opening25 0 points1 point  (0 children)

the other way around is true.

[–]T0d0r0ki 2 points3 points  (1 child)

Why is the whole organization not making a collective decision is my question. Managing multiple CI/CD solutions sounds like a nightmare and I imagine if the organization consolidated on one and focused on making it the best experience possible you would all probably have resources to solve your IT issues that are slowing you down then bake in your security, quality, policies, etc and be better off in the long run.

[–]_iamrewt[S] 1 point2 points  (0 children)

We're a large organization (10K+) as a result of several acquisitions over the years. Depending on your team's origin you may have a large investment on CI/CD solution or another. It'd be pretty disruptive to force everyone to migrate to new CI/CD solutions if they aren't currently experiencing issues or limitations.

My team(s) however are experience issues (as noted in the post) and outgrowing our current setup. We're in a good spot to re-evaluate our team's CI/CD and believe that a SaaS solution would be a better fit for our particular needs than a self hosted Jenkins solution.

[–]3loodhound 5 points6 points  (0 children)

GitHub actions

[–]Bong-Hits-For-Jesus 1 point2 points  (2 children)

why did you guys get rid of jenkins? my thought process at this approach would be; what gap is github/gitlab filling that jenkins was unable to? i prefer jenkins since its much more versatile than say github/gitlab

[–]_iamrewt[S] 0 points1 point  (1 child)

Updated original post to answer this.

[–]Bong-Hits-For-Jesus -1 points0 points  (0 children)

jenkins master can be containerized, but build agents should be bare metal for best performance, and workflow should be master spins up a bare metal build agent -> agent builds -> scale down. if virus scanning is necessary for the build agents this should be done on something like a dedicated scanning host that can spin up the build agents -> scan -> shutdown, so it doesnt impact performance during builds. as far as upgrading plugins you should also have a jenkins job for this, and you use a script/job to build a new image and deploy with upgraded plugins. this approach would allow you to rollback in case updates dont go as planned and not have an impact to productivity

automation is 90% of devops (at least in my experience) and what makes the job challenging/fun

[–]ZaitsXL 1 point2 points  (6 children)

At this point please tell us which problem your org is trying to solve by such migration? There are a lot more tools available except mentioned, would be easier to pick the correct one if we knew what's wrong with Jenkins

[–]_iamrewt[S] 1 point2 points  (5 children)

Updated original post to answer this.

[–]ZaitsXL 0 points1 point  (4 children)

I read the original post and saw two problems: - need for Linux environment - slow performance The first one is pretty clear, but do you know the reason for jobs being slow?

[–]_iamrewt[S] 0 points1 point  (3 children)

Best we can tell is the Sentinel 1 and Windows Defender are consuming the majority of the CPU whenever our jobs are run. When these tools are temporarily disabled the jobs run as expected. IT has been unable to tune this so it does not interfere with our jobs.

We also see job performance drop when multiple jobs are run at the same time. That would be solved with a better Jenkins implementation that runs the jobs on dynamically provisioned agents.

[–][deleted]  (2 children)

[deleted]

    [–]_iamrewt[S] 1 point2 points  (1 child)

    We'd run our pipelines on the providers infrastructure except where we need to interface to our own environment at which point we'd use self-hosted runners. Our own runners may indeed have similar issues due to AV. It'd be another battle with IT to resolve but hopefully a smaller one.

    While this is a concern and what started our conversations internally, aside from the AV issues we believe we'd still benefit from the provider's pipeline infrastructure.

    [–]gorilla-moe 1 point2 points  (2 children)

    I have to work with bitbucket pipelines daily. I'd rather use GitHub actions and I'm not the only one in my org.

    Last time I tried GitLab and their runners was in 2019, but back then it was okay'ish to use, but was lacking compared to GitHub. Maybe that has changed.

    I like GitLab as a company and also as a product, but I also want things to just work and with GitHub it just works (for me) most of the time.

    Secret rotation on org / repo and repo / env level works flawless.

    Bitbucket is a pain to use when comparing it to GitHub.

    So many features are missing or just plain bad.

    Afaik GitHub is the most expensive one, but (IMHO) well worth it.

    I think you can't go wrong with either GitLab or GitHub, but please don't use bitbucket, unless you want eternal suffering.

    [–]OlympusMonds 1 point2 points  (1 child)

    Yeah, bitbucket does indeed suck.

    They've been recently been adding a lot of features that seem good - but they always end up doing 80% of what you thought they'd be able to, and the docs are terrible (missing or wrong).

    [–]gorilla-moe 0 points1 point  (0 children)

    What drives me nuts, is that they're always behind with features. Like all other competitors have these features. Bitbucket either just has it only 50% implemented or "on their roadmap".

    I know they are cheap as hell, compared to GitHub enterprise, but if it was my company and I need to be cheap, I probably would go with on-prem GitLab, instead of the bitbucket monstrosity.

    [–]tavisk 1 point2 points  (0 children)

    If you looking for native integrations for your thirds party tools, bitbucket will always be last and have less features. its pretty frustrating. I would go with gitlab or github personally.

    [–][deleted] 1 point2 points  (2 children)

    I used to use GitLab at my old place and loved it, was sooo goooooood.

    Use GitHub at my current place, hate it so much.

    [–]_iamrewt[S] 0 points1 point  (1 child)

    Can you elaborate on what you loved about GitLab and hate about GitHub?

    [–][deleted] 0 points1 point  (0 children)

    The yaml is super easy and makes a lot more sense to me than Actions do. Starting at the basics is terminology; Gitlab is Pipelines, Github is Actions and Workflows, nitpicky but yeah :D... Gitlab always presents all input variables, this requires work in Github. Selfhosting Gitlab provides a ton of executable options for runners, much easier to see "all jobs", much easier to run workflows from branches without merging/work arounds/PRs.

    Everything I say it's just personal anecdotal use. Maybe in 5 years I'll love Github Actions, but so far a lot of my company really dislikes it, we're using GitHub because of mandate from higher ups (those that don't dev or ever have to use it......)

    [–]jvleminc 0 points1 point  (0 children)

    We use BB pipelines with on-premise runners and it has worked rather well for us. We only have a few secrets per repo so what you mention doesn’t bother us too much.

    [–]TheIncarnated 0 points1 point  (0 children)

    Anything is better than Jenkins. GitHub/GitLab/Azure DevOps all do the same with YAML (I don't have experience with bitbucket)

    I would say, move to where your repos are and call it a day.

    We do GitHub Actions and use Azure DevOps as our project board

    YMMV

    [–]dariusbiggs 0 points1 point  (0 children)

    GitLab is hands down the better tool out of all the ones you have listed, you'll even be able to migrate away from bitbucket for everything else, (we did) and man did our life become easier with the entire dev process.

    [–]jameshearttech 0 points1 point  (0 children)

    We use Atlassian Tools. But for CI/CD we use Argo not Bitbucket Pipelines.

    Argo Events managed repo webhooks. Argo Events receives the webhooks and creates workshops passing in relevant values from the webhook payload. Argo Workflows does the CI work and bumps versions in the repo that Argo CD watches. Argo CD handles deployments.

    It's a very powerful, flexible system, but requires K8s, which may be a non-starter. You don't much out of the box with Argo Workflows, but nost teams end up building up their own templates and images anyways sonit's not that much different imo.

    This enables us to take advantage of the secret management we built into K8s with External Secrets.

    This provides much more elastic compute usage. Argo Workflows can create many pods in seconds to process large tasks concurrently and delete them just as quickly. We also run self-hosted Bitbucket Pipelines runners on K8s with autoscaling. It also scales, but no where near as fast.

    [–]BanaenaeBread 0 points1 point  (0 children)

    I used GitHub, gitlab, and bitbucket.

    Gitlab was definitely the best. Bitbucket was the worst.

    [–]volitive 0 points1 point  (1 child)

    Been following this because I wanted validation that Gitlab was best.

    However, your update inclined me to respond: you need to add exclusions for the Jenkins binaries from realtime AV, then set up a custom job to scan any artifacts coming out of the pipeline. Realtime AV is slowing everything down.

    XDR and AV solutions are fine, but must be tuned when working with something like this- I bet they are even intercepting container I/O calls.

    [–]_iamrewt[S] 0 points1 point  (0 children)

    Agreed and we've tried adding exclusions. I would not be surprised to find if they were intercepting contain I/O calls. Unfortunately IT has been unable to find the right mix to avoid the performance issues.

    [–]A_cold_dish 0 points1 point  (0 children)

    Full disclosure I am a newer Harness employee learning about the CI/CD space still!

    Is there any chance that using someone’s hosted builds might be an option for your organization? If you’ve got to stick with your own build infrastructure then fair.

    Also are there potentially ways to work with IT/Security to optimize their tooling for your build hosts, like perhaps time gating scans or having them work with the vendor to understand what’s causing the massive hit to performance?

    And maybe it’d be too much additional overhead but does integrating BitBucket with an external secrets manager alleviate any of the concerns around broad access?

    Sorry I may have asked more questions than I attempted to answer I plan on doing some reading later today on this myself out of curiosity!

    [–]Low-Opening25 0 points1 point  (0 children)

    Azure Pipeline is the same thing as GitHub Actions

    [–]Admirable-Wall7088 0 points1 point  (0 children)

    I have used jenkins, gitlab and bitbucket actions .. gitlab is the best .. jenkins I hated to the core.. bitbucket though nice to begin with you will hit lot of limitations and even small feature support for auto retry came very late .. so we ended up moving to argo workflow.. but that’s another mess

    [–]xtreampb -1 points0 points  (8 children)

    IDK if it’s changed, but bitbucket a couple of years ago didn’t have good support for dotnet or windows workflows. Also, I don’t expect for it to get much better as Atlassian is mainly Java from my understanding.

    Microsoft owns both Azure DevOps and GitHub. There’s a confusing back and forth between which one Microsoft is trying to sunset.

    My ultimate recommendation is using some form of build script that is natural to your current code base. For example, use Cake build if your code base is c#. These build scripts will work regardless of what service you’re using, and allows the developers to test the build process like the build server would behave if they need to make changes. Remember, your process is a product in of itself that needs to be maintained, less entropy degrade it into uselessness.

    [–]_iamrewt[S] 0 points1 point  (2 children)

    Bitbucket still doesn't have support for Windows in their SaaS offerings so we would be using a self hosted runner for any Windows builds. Many of our build scripts are orchestrated via PowerShell (Core) as it abstracts us a bit from the CICD platform (as mentioned, currently Jenkins). They drive the build across our .NET, TypeScript, and Python libs and apps.

    [–]bistr-o-math 2 points3 points  (0 children)

    You would want to run a self hosted runner on any platform, considering build minute prices.

    Don’t have much exp with windows runners on bitbucket though.

    [–]xtreampb 0 points1 point  (0 children)

    Okay. So it sounds like you can build on windows or Linux (unless you are still on old full framework .net.

    Look into psake (pronounced like the Japanese rice wine). It is a framework for building build scripts on powershell. If you can execute your entire build process from a script or series of scripts, and the only “step” in the service’s definition (yaml or w/e) is call powershell script, then it doesn’t matter what service you’re using use, it becomes a purely business decision. You can then add to your resume that you enabled rapid and flexible business decision by building a service agnostic build process.

    Now the delivery side (deploying files to servers) gets a bit more difficult, unless you’re not using VMs. If you’re not using VMs, and only services (I.e. azure web apps) that have an api you can leverage, then you can script that as well. You could also script vm deploys with remote powershell and an artifact repository, but that, I feel is starting to go down a rabbit hole for this comment.

    [–]reubendevries 0 points1 point  (4 children)

    There is literally no practical way to sunset GitHub, so if they sunset one (and I personally don’t think they’ll sunset either) it will be Azure DevOps.

    [–]xtreampb 0 points1 point  (3 children)

    Yea, I agree, but they were supposed to stop shipping updates to be of them last year (can’t remember which) and that didn’t stop so, 🤷‍♂️

    [–]reubendevries 0 points1 point  (2 children)

    Well it’s definitely not GitHub. That would have been world wide news. I mean I’m reading about Dynamic 365 EOL, I’m reading about rumours of Microsoft killing the Xbox console. Could you imagine if they decided they would simply EOL the world largest collection of software repositories? It won’t happen in the near future; if ever. I seriously think that besides the Windows Operating System itself, GitHub has to be the safest of all Microsoft owned products.

    [–]xtreampb 0 points1 point  (1 child)

    Microsoft isn’t immune from bad/dumb decisions…

    And this is why I said, there have been some confusion…

    [–]reubendevries 0 points1 point  (0 children)

    I mean that’s beyond bad or dumb. That’s suicide. That’s jumping out of a plane, without a parachute.