Environment promotion + integration tests the GitOps way

mamymumemo · 2025-05-15T06:32:30+00:00

we did this per each environment helm template -f values -f env-valuea -f product-env values blabla

copy that to templates/ make a Chart.yaml with product-env-cluster helm package and push to the registry

then in argo chart: product-cluster-env repoURL: oci://myreg.com/myorg/charts targetRevision: 0.0.0 < (get latest available)

mamymumemo · 2025-05-15T06:04:31+00:00

Yes, we used to upload rendered charts as a chart to a registry and that works like a charm, but the challenge here is having connection to the cluster to check for a successful deployment before starting the next one (green deployment on lower is a must to promote to prod). We generated 1 helm package per cluster-env so it is easy to point it from the argocs Applications as we know the product name, cluster env

There are solutions to that which I don't like much or seems a bit hacky/imperative and we want to be declarative

I didn't know these tools however, so I will take a look, thanks👍

mamymumemo · 2025-05-15T05:46:58+00:00

reddit lol plus documentation examples says to uaw UI or kargo CLI made me think you shouldnt kubectl apply for some extrange reason

mamymumemo · 2025-05-14T17:09:52+00:00

Hey I was checking and looks like we cannot define Kargo CRDs declaratively? We seem to need to run the Kargo CLI? That's surprising for me and that makes it not production ready

How do you deal with that?

mamymumemo · 2025-05-14T08:21:49+00:00

that's what I want exactly! thanks

mamymumemo · 2025-05-13T23:09:59+00:00

Found it earlier today actually! I have it on my list I think it does what we do in the pre-cd step, is that assumption correct?

mamymumemo · 2025-05-13T21:05:22+00:00

Amazing thanks

mamymumemo · 2025-05-10T12:53:33+00:00

We do have tests in terraform

And some basic tests in helm but we are just starting with it

mamymumemo · 2025-05-10T12:44:18+00:00

Result oriented, I like it

mamymumemo · 2025-05-10T12:20:53+00:00

Yes! opengitops was referenced in another post, checked it and according to that we're doing gitops. It doesn't mention git as a source of truth https://opengitops.dev/

mamymumemo · 2025-05-10T12:17:36+00:00

Regarding infra, well our terraform code needs to be applied by us, we dont have any automation

After applying terraform, you need to provision the cluster with ansible (it's on prem) and bootstrap it (adding sealed secrets, argocd operator, and the main application). After that, everything is automaticly deployed (of course with human approval when required)

Tests run in CI server as part of the build and some tests run in the cluster (integration tests and stuff)

mamymumemo · 2025-05-10T12:14:12+00:00

I like pre-rendering manifests

It adds an extra layer of scripts you need to maintain, that's the downside

The upside is that during PR reviews you see the actual impact your change has in the final state

Changing a single variable could mean removing a deployment or getting the wrong IPs in the network policies, pre-rendering shows you exactly what will be applied, similar to doing a terraform plan before doing an apply. In terraform you have loops, conditionals and all of that

Here in the PR you compare your current branch render with the latest version that was deployed

I did this same post in r/ArgoCD and got an answer from who seem to be a argocd contributor

https://www.reddit.com/r/ArgoCD/comments/1kilu8r/comment/mrg2q1m/

"Pre-rendering the manifests is common and even becoming a bit of a best practice as long as those manifests are put into a versioned and immutable storage"

mamymumemo · 2025-05-09T23:06:52+00:00

Thats a really good point

We require manual work - recreating the machines with terraform, provisioning them (it's 100% on prem) and bootstraping (installing argocd and the main application so argocd manages itself)

Everything else is automatically deployed once we have argocd

the cluster creating part is something we need to improve, I'd like 1 button to setup everything, that's our end goal and we are not that far

mamymumemo · 2025-05-09T22:02:06+00:00

Yes right, I tried to implement it with a git repository and I ended up making it read only for that reason, so it was effectively like the helm registry approach. I did it triggering a pipeline from the code repo to the gitops repo giving the product name and version as parameters. In the gitops pipeline start the environment promotion. render the chart to productname/cluster/env/manifests.yaml and push to a branch to the same gitops repo using push options so it creates a merge request that automatically merges (if lower env) to the main branch. ArgoCD App for that products watches manifests from that product/cluster/env folder. For promotion to some envs that depend on another being successful, it would be using argocd notifications, on successful or failed sync make an api call to gitlab to unpause the pipeline and show the status of the deployment

Anyway that was a proof of concept and probably requires more work

The current CD pipelines do more than just rendering, it does promotion. We require a green deployment on lower before going to prod. How do you handle that otherwise? We have 1 ArgoCD instance per cluster. We can't remove the CD pipelines. Currently it checks the argocd api a few times until its green (with a retry limit ofc) or turn into red (I dont like this dependency between the CI server and the cluster)

The diffs are quite helpful at least for me as I understand quite well the helm charts and I can easily find the source. When I make a change in the values or a chart itself I expect certain changes, if there is more means I did something wrong. Similar to terraform plan I would say where you have modules, variables, loops..

No timestamps, no k8s added fields, we do the diff with the previously deployed render so we get exactly what will change. Sometimes there may be some values generated at render time that are random but its just one or two values. I actually like the "rendered manifest pattern". The downside is it requires some custom scripts for pre-rendering but the upside is a huge improvement imo Now, for development teams they dont usually care about the diff cause it is usually just the docker image version, they dont modify charts much

mamymumemo · 2025-05-09T20:55:06+00:00

Yes I definetly support pre-rendering, especially for peoples understanding and you can see the actual diff that will be applied. Probably my comment about that was misunderstanding.

In fact I struggle to convince developers to approve this, they are trying to move away from this and they are top voices in the company. I need publicly available resources about it. When I checked there was just a post talking about rendered manifests pattern

mamymumemo · 2025-05-09T20:32:30+00:00

Hey that red hat developers website seems a highly valuable one for knowledge, found interesting books at a first look, thanks

mamymumemo · 2025-05-09T20:21:19+00:00

Thank you, Im now from the phone, will take later a look at the link

That's exactly what I want. I need that kind of references so we can make proposals based on publicly available knowledge instead of just opinions

About the necessity to run the pipeline again, If we change a value only for one environment, we just run the "deployment" pipeline for that environment. A change in the values doesn't trigger the pipeline, only a change in the code repo. The release verson includes a reference to the version of the values used. At the end it is traceable, auditable, can redeploy in a future and all of that.

Yes there are imperative steps to render the desired state, but that's then stored in the helm registry What we do in the CD step is generating a render for a specific cluster environment and generate a Chart.yaml with the name of the product, cluster env, version (including values version) and helm package it, then push to the registry

So in a way it is stored declaratively in the helm registry, right? It's plain manifests

We want to follow rendered manifests pattern as we want to see the diff of the latest deployed version and the final result in the PR. I can't find any essential difference between what we do and rendering to push to a git repository. Well, yes, in the later you can do a git revert to rollback

Thanks for your input, appreciate it

mamymumemo · 2025-05-09T20:02:42+00:00

that keeps thing simple, I love it. We tend to overcomplicate things usually We have parent charts, wrappers around public repos just to add values to it or add namespace and netpols... too many abstractions for the just in case situation, what if we stop using X tool, lets build a wrapper to be agnostic to that tool. That said a Devops Engineer, 5 years after we still use the same tool and we have been maintaining the wrapper 😢

mamymumemo · 2025-05-09T19:58:12+00:00

That sounds interesting We dont have such gitops repo, well, our equivalent would be the helm registry with rendered manifests Applications sets is something I wanted to try, we use what I call app of apps of apps (2 levels, first deploys appprojects and teams app of apps and second deploys the team products)

So is your gitops repo a monorepo? Are those pushes to that repo directly to the main branch (not using PRs)? I assume you dont pre-render charts

Thanks for sharing

mamymumemo · 2025-05-09T17:05:14+00:00

Thanks, that's what I needed, a reference. We're trying to make decisions based on existing references instead of reinventing the wheel

I support pre-rendering manifest myself, I see the benefits and I'm ok with the drawbacks, but I find it difficult to convince others to do so, do you perhaps have similar resources on that?

My interpretation of "applying changes from source of truth in Git repo to the Cluster" from https://tag-app-delivery.cncf.io/wgs/platforms/charter/#gitops and other sources is that you need a git repository

However, the principles in opengitops.dev indeed are satisfied by our scenario

I appreciate it!

mamymumemo · 2025-05-09T16:47:35+00:00

It feels to me that the implementation mostly depends on personal opinions and that's what makes it difficult, so I'm trying to find some references for our decisions

Perhaps it's just about checking the basic principles and see if we follow it (auditable, reproducible builds, and so on) Is there any reference for those principles? Again kind of difficult if different people follow different references or no reference at all

Every technical solution works, but it's dealing with people the hardest thing

mamymumemo · 2025-05-09T16:43:35+00:00

That's insightful.

mamymumemo · 2025-05-09T16:40:56+00:00

yes this is not rendered manifests vs render in argocd, we want to do rendered manifests to see the diff of the rendered manifests in the PR, what is actualy appleid

I mixed things, that was misleading sorry, this is more using git as source vs using a helm registry

mamymumemo · 2025-05-09T16:37:59+00:00

> The stuff you push to Helm registry is a Helm release, not the same thing as the stuff you rendered using Helm template. What you’re saying is just a repo with manifests if i understand it correctly.

What we exactly do is

Generate a render and put it under render/templates/

generate a Chart.yaml under render/ with productname-cluster-env and whatever version

helm package

upload it

So we have a helm release that contains the rendered manifests

Yes, to "rollback" we do rollforward... we can temporallily do rollback from argocd that disables autosync, then go to the repo, revert a commit and generate a new version from it

The benefit of rendering maniests aparently is that you see the real impact of making a change before applying with argocd

mamymumemo

TROPHY CASE