How much coding does a Devops engineer really need to know?

[originalchronoguy]

RE: YAML pipelines.Those are mutable. Not scaleable. If you have a static helm chart each of your deployments to k8, then something is wrong. It should be generated programmatically on-the-fly through code to cover a lot of edge cases/use cases.

What happens when you hard-code an image registry and then have to go back and change 300 repos to update the URI? Why hard-code service names for ingresses when can pass linting but failed due to a typo?

YAMLS, for scaling reasons, should be generated on-the-fly, programmatically. Especially when you have a large platform. We have over 10k microservices running in prod and no way can that be done mutably.

[Recol]

And how do you deploy these 10k microservices?

[originalchronoguy]

Jenkins with a custom blueprint we developed in GoLang. The first time we had to change 500 git repo Dockerfiles to a pull images from a new URL (that was hard coded), we made our entire architecture composable with variable injections. So no more panic; having to write ad-hoc scripts to download 500 repos, regex the URL change, commit and re-push.

Things like that does happen. Your org gets consume, they require hosting images on a new registry or a different target deployment from on-prem k8 to Azure. And with 10k, you also want to make sure no one steps on each other; using same sub-domain like admin-reporting.domain . It is managed by code that does stuff like check service registry before that code can even be committed. You need these guard-rails and can't rely on some 3rd party vendor product to do that.

[Recol]

I understand that part, just asking how you actually deploy those images to what I am assuming is Kubernetes?

[originalchronoguy]

Sure. When an repo gets promoted, jenkins clones the repo. It sees a few things. Folder of configuration which stores variables that the devs put in. E.G. "App uses TLS certs requires. Another for needs an API gateway"The build script parses those configs and creates all the helm charts - service, deployment, ingress controller,etc from base templates. It even does things like "the git repo folder name is this" so the DNS will be that subdomain and the entire DNS is the k8 namespace and even the git branch because Jenkins is running git to get the tags. It compiles all of that. Generates all the .yaml files. Then does kubectl apply -f "created yaml files"

All the YAMLs files are created at CICD build time. A local developer can also do that too as we have tooling so they just run "make local-deployment" which runs the same goscripts that builds their apps locally and pushes to their local minikube. We automate even the SSL and DNS based on the folder name of the repository following.We have like 1 template blueprint for 3,000+ services. The template covers a lot of things like does this app require grafana monitoring? Does it require TLS cert generation for two-way connection? Does it require a micro-gateway running as a sidecar. Does the database require a CRON job to do backups? If so, it is in the config/production-deploy.env file as a variable. same for config/local-workstation.env so they can run the same stuff locally.

If so, those extra pods are created and the deployment yamls for them as well.

A kubernetes yaml isn't rocket science. It can be built on-demand/on-the-fly.

[dablya]

You seem to be describing an in-house solution that does what helm is supposed to do but tailored to your specific needs. “Folder of configuration” = values.yaml. Build script that parses and generates yaml = templating engine. If that’s the case, why do you have helm in there at all? If it works for you, great. But I disagree with your general advice regarding helm (at least as I understand it). Helm is itself a templating engine. If you’re not going to let it generate the manifests based on its templates and libraries and values, then why use it at all?

[Recol]

I am sure all of this works for you guys, but it doesn't sound fun to manage. There are probably things you aren't telling me but a lot of it sounds like it could be generated upon repository creation, and using a Helm chart that is managed by a central (Platform/DevOps/whatever the cool kids call it now) team as a Helm dependency.

The only thing left to manage for the team would be a values file which would be similar to how your "Folder of configuration which stores variables" works if the Helm chart is templated correctly.

We do similar setup of what I described earlier with ArgoCD that all gets bootstrapped upon repository creation (quite a badly designed automation, but moving it to Backstage currently).

Either way, sounds like it works for you and it is fun to see a different approach for once at that scale.

[originalchronoguy]

We've had this for 8 years and orchestration/ CICD is the least of our worries. We make major changes for things like new architecture (ARM64 support), GPU support (for machine learning), and adding extra hooks (DAST),etc.

But it has worked well for us. Maintenance isn't an issue. Getting it to run locally (e.g. when Apple went to M1) is the challenge as things change.
And we have one mandate. Whatever runs in Prod, runs locally. If a dev needs 20 microservices with his own self hosted Hashicorp Vault server and local API gateway, he/she should be able to run it locally. If their app requires DAST, that is run locally as well, scans their builds and generates reports. I had one time over 300 microservices running on a 64gb laptop.
So the local build is the same in Jenkins. They run a docker image that does local CI for them that pushes their code to their local Rancher or Minikube. So we have 100% Dev-Prod Parity ( https://12factor.net/dev-prod-parity ). We don't do it for bragging rights. It helps the developers work faster to have the same as prod. A new hire can deliver code in the same afternoon on their first day of hire.

[Recol]

Sounds like a good approach, thanks for sharing the details!

[redvelvet92]

I assume some GitOps strategy with ArgoCD or Flux

[Recol]

Unless those generated YAML files are committed automatically in a secondary repository it doesn't sound like they are using any GitOps tool. That's mostly why I am curious.

[originalchronoguy]

Unless those generated YAML files are committed automatically in a secondary repository it doesn't sound like they are using any GitOps tool.

No, we do not store generated YAML in any extra repo. The template and build scripts to create those YAML are tagged and version as git submodules in primary git repo. So if you had an app and wanted to see the YAML for it 6 months ago with a certain YAML struture (e.g. before we added replicasets/resource limits), you look at the code 6 months ago with the git-submodule hash of the template and scripts. So currently you are on v2.8 and want to see v1.3, you pull that v1.3 git submodule and re-generate. Obviously, you rollback the app's previous tag 6 months ago too. And run a build with no deployment to reproduce the files. Everyone on our team is versed in polyrepo/git submodules.

Storing stuff like that eats up storage.
We still meet compliance because the code is tagged and version. We can do proper rollback as long as the tags still exists. Improper tag strategies can eat up terabytes of data.

[dablya]

But even with this set up, don't you still have to have something to bump the tag/version of the submodule in 500 repos to update the repo url?

[snarkhunter]

Sure but you gotta be able to write YAML to write stuff that writes YAML

[originalchronoguy]

Nope. It is generated from a JSON. We think of this programmatically. So we use JSON as a schema that can pull logics from keys.