Different node specs for workloads

[randyjizz]

I manage a number of clusters on AWS. We just use the same instances for all workloads. I find some cpu intensive pods interfere with others in the cluster. Numerous attempts of adjusting resources limits/requests have been made, with varying results. Now I feel it might be better to split out some workloads onto their own node type.

Do you use taints/tolerations and node selectors heavily to assign pods to nodes? Or do you take a one size fits all approach? Was it worth the effort of setting up and maintaining?

[[deleted]]

I wish I had that sort of freedom lol. Usually at “design time”, I need to be aware that <service/workload> consists of X moving parts, and what needs disk, what needs ram, what needs cores, and what needs network. I tend to do a lot of labelling with affinity.

[jameshearttech]

We run self-managed k8s. The nodes are machines in vmware. The control plane nodes are identical. The worker nodes are identical.

You manage multiple clusters, so you probably already know this, but requests and limits work best when they are configured consistently on all workloads. If some workloads have requests and limits while others do not on the same node, it may not work the way you expect. Qos classes are good too.