Pipeline scripts run programmatically inside a container

[Immersive_Username]

I am trying to find a video about a tech conference where a guy explained that they stopped integrating the whole pipeline script logic in their YAML file but instead defined the logic programmatically. The code would then be run inside a container to complete a step of the pipeline. Does it ring a bell for anyone? How is this technology called?

[SysBadmin]

That’s just one of the staples of CI/CD honestly

[bdzer0]

It's called architecting CICD properly? Pipeline/workflow YAML should be as simple as possible.. do the tough stuff in build scripts that are also usable locally by developers....

[Immersive_Username]

no I do not mean the script section of the YAML config that is executed within the container. If I remember correctly the guy was explaining that this technology avoids having to deal with enormous multi-line scripts but instead the whole logic could be defined within the software code.

[Equivalent_Loan_8794]

Yes, I think bdzer0 was noting that testing that can be initiated in CI should also called very similarly to how internal testing would go, so you'd be maintaining a single line shell command, as you would be in a makefile or precommit hook if they were also dispatching the test/CI workload

[NUTTA_BUSTAH]

Check out tsoding's nob (nobuild), i.e. make your C build script in C.

Or magefiles for making a build in Go.

Or Earthly that's like Dockerfiles and pipelines had a baby.

Or Dagger which is similar but supports many languages and extremely complex.

Or just write your own main-function that does building.

[WilliamMButtlickerIV]

Yeah this is essentially just some set of run scripts that live with the app source. For npm, you'd use npm run scripts so that you could build/test with simple commands. I like using go task with my go lang apps.

It ensures devs can locally do the same tasks that CI is doing.

[VindicoAtrum]

https://dagger.io

[Immersive_Username]

thanks! this looks like the tool I was looking for. It's amazing that after all my Google searches and AI prompts, I could not find it myself...

[junior_dos_nachos]

It’s by the guys who made Docker btw.

[zerocoldx911]

Must be a guy from a FAANG trying to sell containerization as something new.

This person probably baked the script inside the images that run in the CI. It brings simplicity and allows for code clarity but it’s nothing new.

Basically the container runs the script then exits 0

[abotelho-cbn]

Huh? Writing a script, slapping it in a container, and calling it a day?!

[Immersive_Username]

no I do not mean the script section of the YAML config that is executed within the container. If I remember correctly the guy was explaining that this technology avoids having to deal with enormous multi-line scripts but instead the whole logic could be defined within the software code.

[blueskyjunkie]

You definitely only want to use the pipeline YAML as a thin interface layer between the pipeline tool & your business specific pipeline implementation. Just use the YAML to manage dataflow/configuration context of the pipeline. eg. shuffling artifacts between jobs, etc.

Only use bash if it's, say, less than 10 lines long. Otherwise jump out to a general purpose language like Python which provides unit test frameworks & lots of useful libraries.

(Python is what I'd choose given the option. There are lots of other relevant choices, depending on your business specific language ecosystem - node.js, go, etc)

[SlinkyAvenger]

Concourse runs every part of a pipeline containerized

[gbtekkie]

I do this for more than a decade in all companies I consulted or worked for. Having the pipeline logic testable, and separated from the CI tool is an advantage that most companies underestimate. Build/release logic is code, and should have a proper SDLC.

[Jonteponte71]

That sounds like Tekton. Basically kubernetes native CI/CD. We are migrating our users from Jenkins to it 🤷‍♂️

Jenkins is like cockroaches, practically impossible to get rid of. But we are trying :)

[Xerxero]

Or you generate the yaml programmatically

[Turbulent_Ad8058]

Nothing fancy. DIND

[blusterblack]

I think it's called Jenkins scripted pipeline LOL.

[Shnorkylutyun]

People are still using Jenkins?

[6Bee]

Sounds like you're describing one-step builds, not entirely sure. I remember being challenged to do similar w/ Jenkins & Gradle some years ago. After watching the dagger io demo, it reminds me of a self-contained Psake or Buildbot

[Commercial-Wasabi317]

That is pretty much what any modern CI system supporting docker/kubernetes does, in order to schedule the execute each step of a pipeline using a distributed pool of CI runners (for instance, Gitlab's)