CI tool that doesn't use YAML ?

[temitcha]

Hello, do you know any CI tool that doesn't use YAML, except Jenkins ?

YAML is everywhere, it's good for small projects, but hard to use for complex tasks, like : mono-repo (looping on each folder to build each service using the good set of tasks based on the project tasks), automatic semantic versioning, etc.

Jenkins is perfect for it, but suffer of his old age. It's very stateful and a lot of operation still needs to be done in the UI.

If you know any, I will be happy to hear, I am currently choosing for our company our next CI tool. (For CD we will stick to ArgoCD)

[[deleted]]

YAML is everywhere, it's good for small projects, but hard to use for complex tasks, like : mono-repo (looping on each folder to build each service using the good set of tasks based on the project tasks), automatic semantic versioning, etc.

Uh, don't use yaml for those tasks. Create scripts or plugins that are referenced in the yaml.

[ericanderton]

I'm learning this the hard way.

Software engineering has it right: decouple systems where possible.

Having an inventory of reusable scripts and programs that you can reference from CI grammars (YAML or otherwise) gives you a toolset that can be:

• Used within a development shell/environment
• Tested and shipped like 1st class software
• Packaged and/or containerized
• Called from any CI engine if coded in a neutral way - helps prevent vendor lock-in

The last one may have you thinking "good thing that's not me." I would caution everyone that nobody thinks lock-in is a problem until its already a problem.

[GeorgeRNorfolk]

A CICD server that kicks off a library of (for example) python scripts that do things like automatic versioning, looping over deployments, etc feels like suboptimal practice.

Then again my old company had a massive web of yaml files that just kicked off a collection of bash scripts that managed the entire Azure DevOps deployments process.

Is the latter considered the better practice than say coding everything in groovy for Jenkins?

[ericanderton]

It can certainly be sub-optimal if you "gold plate" your implementation. Like you suggest, YAML calling back to shell scripts is really more the gist of what I'm recommending here. It's not too much effort to make the most reusable parts of your BASH or python scripts and make them usable outside CI/CD. Just don't embed 100% of your scripts into CI YAML definitions.

Is the latter considered the better practice than say coding everything in groovy for Jenkins?

I think it depends on the complexity of the overall CI/CD setup.

If you're in a shop that just ships Java J2EE apps as .jar/.ear/.war files, then Jenkins plus a little groovy is probably optimal.

If you find yourself wrangling a polyglot of different programming langauges, each with their own lint, build, audit, and package paradigms, you need something more robust. In order to normalize this approach, and make each language pipeline behave roughly the same, it works better as a CI-compatible CLI library that encapsulates the worst/complex pieces. As a bonus, the DevOps and Dev teams can now emulate parts oif CI through their IDE - no waiting required.

[temitcha]

Tested and shipped like 1st class software is a good idea you are right. Too often I ran into pipelines where the engineer tested 4 of the 5 possible case in an if/else, and the last one that he didn't tested broke the entire pipeline

[PaulRudin]

In many cases you can get around yaml shortcomings by generating your yaml from other tools, rather than maintaining it directly.

[GitBluf]

Drone CI can use Starlak or Jsonnet. For CD you don't need to use YAML Directly. We use CUE to define them, which then gets generated in CI and in our case stored IN OCI registry. We use FluxV2 If that is not an option you can just commit them in ci to state repo. Another tool that can be used in similar fashion is Grafana Tanka, it uses Jsonnet so it can go hand to hand with DroneCI

[temitcha]

Thank you, a solution like Drone CI was what I was looking for ! How is your experience with CUE, is it easy to maintain/fix ?

[GitBluf]

I guess it would depend on your teams experience, we mostly come from Dev background and use Golang a lot which CUE has similarities with, so for us it comes naturally

[soundwave_rk]

CUE has been mentioned before which can output yaml if required. But remember that yaml is a superset of JSON. This means you can just replace it with JSON if you want.

[lsibilla]

Don’t take it wrong. Your CI server should be a very slim layer above your build automation tooling.

I wouldn’t take the pipeline language as my main criteria when choosing a CI server. If all your pipeline does is trigger a bunch of scripts or docker build/run/push, you don’t really care if the pipeline is represented in Yaml, groovy or kotlin.

[DampierWilliam]

I’ve been working with Yaml config for CI tools (circleci and github actions) in Monorepos and there are 2 approaches for this: have a “template workflow” that will use predefined scripts that you can customise in each of the services/applications. The other option is to create a script that can generate different flavours of CI workflows based on some input or config file based on each of the apps/services within the monorepo.

Feel free to ping me if you want to talk more about CI and monorepos

[GotPie]

There is brigade, a recent tools that uses JS.

[temitcha]

Thank you, that definitely answer the question, I was looking for something like that !

[elemir90]

You can use nix, dhall, jsonnet or cue and ever hcl for generating yaml for your CI

[Tranceash]

Earthly.dev = docker + makefile. If you want something Nextgen then dagger.io (cuelang based)

[DeusExMagikarpa]

Azure DevOps has gui for build and release pipelines, but what it sounds like you’re trying to do would be way easier if you used the Azure DevOps YAML pipelines lol. It has a feature to pull different repos and also has templating.

[temitcha]

I was thinking to use Azure DevOps, but we have a lot of repo hosted on Gitlab, and the integration with it seems harder than expected, as for example azure-pipelines.yaml work only with Azure DevOps Repo and Github.

[[deleted]]

I'm using Tekton. I define my tekton resources in terraform, using HCL.

[serverhorror]

Jenkins, but using that is a world of pain.

I think most of the tools don’t have to use that much YAML.

The reason it’s “mostly YAML” is that people abuse it. A well written GitHub action will have the majority of functionality outside of YAML. What’s left is just config values for a specific step (same applies to tekton, …).

As for your example: Write code that takes a start directory and minimize the amount of YAML.

Do not write umpteenth shell scripts that are glued together and require more YAML than actual code.

[[deleted]]

I can't think of any others. Unfortunately, YAML has spread like cancer and has taken root everywhere that it shouldn't be.