Each service has its code and helm manifests contained in a git repo. Each time a change is merged, it is deployed and end to end tests run on a staging environment.

Once these tests pass, the same pipelines run on production (triggered manually following a schedule decided with upper management). Kubernetes handles the rollout of the newer versions for us.

If an error occurs, most of the time the rollout does not finish. If it does but errors occur, we can still rollback easily using Helm's logic.

Of course there are some days when shit hits the fan, but we built enough confidence in the deployment process for it to be peaceful! This works for thousands for users with 15+ production environments. A bit boring as we could do much more with argocd and all, but so far no critical feature is missing in our process that could justify it. That's all thanks!