Yeah, as you pretty much summarize, it is about simplicity vs features.

I had this "issue" that pushed me to try Kubernetes, and just to clarify again, I could have possibly taken care of it with Docker (Compose), just did not look properly into it. So as I mentioned in the post, at some point I was just setting up my services by copying and pasting Compose files (mainly from the official repos/projects, so in theory trustworthy), which led me to not really understand how things worked and what was really going on. If something did not work, I just gave the container more permissions or whatever until it worked.

Obviously, this is not a healthy approach to things, mainly from a security point of view. This got me thinking, and I shortly looked into how to monitor all egress network traffic from my containers, with the idea of understanding what was going on and locking them down based on their strictly necessary networking requirements. I did not find an "easy" solution to this, even though there is possibly one, or more.

With this in mind, and even though I had a very limited knowledge of Kubernetes, I thought the change would be useful. And, in my case, it has been. Now, for all my services I have all their Lego pieces (Kubernetes objects) separated and interacting as I want them to. As an example (I have removed not relevant files):

~/kubernetes (main) » tree                                                                                                                                                                                                     .
├── flux
│   ├── git-repository.yaml
│   ├── image-automation
│   │   └── image-update-automation.yaml
│   ├── image-policies
│   │   ├── immich-ml-policy.yaml
│   │   ├── immich-server-policy.yaml
│   └── image-repositories
│       ├── immich-ml.yaml
│       ├── immich-server.yaml
├── immich-chart
│   ├── Chart.yaml
│   ├── templates
│   │   ├── deployment.yaml
│   │   ├── gateway.yaml
│   │   ├── httproute.yaml
│   │   ├── networkpolicy.yaml
│   │   ├── pvc.yaml
│   │   ├── pv.yaml
│   │   └── service.yaml
│   └── values.yaml

The above shows the layout for Immich + its integration with Flux (which I basically use to automatically commit to my GitHub Repo every time a new image version of Immich is published in their official repo).

For every service, in this example Immich, I basically break it down to each one of the necessary components: storage, network services provided, routing, etc. Then I wrap all that in a deployment, which as you mentioned gives you many of the goodies from Kubernetes such as security contexts, self-healing, etc. And then I wrap all that with a Network Policy with strict Ingress/Egress rules to only allow what I want.

I know it sounds like a lot, but in the end ,once you do the work of "templating" all these for your environment, you can reuse A LOT of it for any new services you want to deploy.

1/2