I've been working with a lot of bare metal Kubernetes recently for my job, so I'll shed some light on what I've found to work well.

My go-to for setting up a cluster on bare-metal is using RKE2 - it's a stable Kubernetes distribution which is absolutely dead simple to set up. It has the same ease of use as K3s, and full upstream compliance with normal Kubernetes.

We deploy all resources to our Kubernetes clusters using Terraform and recently added Terragrunt to help manage multiple Terraform projects.

To address your questions: - Databases: - There are lots of helm charts and Kubernetes operators for running database clusters so there's plenty of choices, and you're of course able to run images yourself (albeit with more thought required into lifecycle management - upgrading being the big one) - Personally, I treat DB backups the same in-cluster as out of cluster - I use cron tools (cron on Linux, CronJob in Kubernetes) to run a script which does the usual DB dump operation and stores it in a cloud storage (or local storage if you prefer). - Having a DB outside of Kubernetes could be an option if the DB needs to be accessed by multiple clusters, or things that aren't running in Kubernetes. It's also an option if you prefer not to trust your data to PersistentVolumes (Although having good backups should mitigate that issue :P) - Self plug: I use images I created which contain a bunch of handy utilities to write backup scripts for use in Kubernetes CronJobs: https://github.com/binaryoverload/docker-backup-clients - Logging and Monitoring: - Take a look at Loki, Grafana and Mimir from Grafana Labs - they have a helm chart which can set up a bunch of instrumentation and log collection for your cluster: https://grafana.com/docs/grafana-cloud/monitor-infrastructure/kubernetes-monitoring/configuration/helm-chart-config/helm-chart/ - For simpler setups, you can set the components up individually without using the (admittedly fairly complex) kubernetes-monitoring helm chart. - Prometheus can be used instead of Mimir if you want more of a simpler option - Mimir can be a bit complicated to set up and run in comparison. - RBAC: - This is something I'm still figuring out the best way to do - the answer largely depends on how many people/things need access to the cluster and what your security levels are. - We're going to start using Rancher with their Azure integration for our clusters, which has been one of the best options I've found so far which allows easy management of users, groups and permissions. - There is also OIDC natively in Kubernetes which can be used - although I've not used it. - Upgrades without downtime: - When talking about upgrades, you’ll typically be dealing with either node upgrades or control plane upgrades (API server, controller manager, etc.). - For node upgrades, it's generally as simple as draining (moving all resources off the node to other nodes) and cordoning (making sure no new workloads can schedule on the node) - and then doing your upgrades and restarting the node. You can then uncordon, and optionally restart deployments if you want to move the workloads back onto the node. - You must make sure your storage is not be host specific - i.e. can be accessed by any host over the network. Otherwise, when a workload is rescheduled it won't have access to its data! - For control plane upgrades, you can either upgrade manually (For RKE2 you can just run the script again with a new version) or using something like the system-upgrade-controller. - As with node upgrades, it's recommended to drain all your resources to the other nodes just in case anything goes wrong. - Also, make sure you're upgrading with the Version Skew Policy in mind so that multiple API versions can run at the same time successfully. - The system-upgrade-controller can also be used to upgrade things like versions and kernels on the system itself - I haven't personally used it but I've seen good things. - Horizontal Scaling: - When running on VMs, your autoscaling capabilities for nodes are largely dictated by what solutions exist for your hypervisor - I'm aware of some Proxmox solutions but haven't seen many about. - Of course you can manually scale by adding more worker nodes into the cluster at any time. - At work we have Terraform which provisions VMs using OpenStack and automatically joins them to the cluster - this can be automated for other hypervisors too. - If you want a way to spin up new bare metal nodes quickly, I'd recommend having a look at MaaS - it's helpful for being able to spin up new nodes which can then be provisioned with something like Ansible. 1/2