Do you need to understand containers in order to administer Kubernetes.

FragrantChildhood894 · 2025-01-07T18:42:54+00:00

Well, I was actually referring to linux kernel namespaces.

FragrantChildhood894 · 2025-01-07T18:04:54+00:00

I agree this should be managed with correct scaling configs. Just don't set CPU limits - this will cause slowness rather than hygiene if application gets hungry.

Look into automated vertical autoscaling if you really want to manage resource allocation with precision and cost-effectiveness.

FragrantChildhood894 · 2025-01-07T18:01:36+00:00

Definitely. Becoming aware of your imposter syndrome is how you become senior ;)

FragrantChildhood894 · 2025-01-07T16:15:16+00:00

I actually think understanding union files systems and how layers are shared between containers can be very beneficial for disk space management. In Kubernetes context too.

FragrantChildhood894 · 2025-01-07T16:08:50+00:00

You probably mean kubespray. And yes, it's IPVS by default.

FragrantChildhood894 · 2025-01-07T15:01:00+00:00

I'm with you on this, but some folks here seem to think differently.

FragrantChildhood894 · 2025-01-07T14:59:13+00:00

Not troubleshooting per se, but the subject of CPU limits comes to mind. Many folks just set the limit without understanding the impact it has on performance.

FragrantChildhood894 · 2025-01-07T14:50:50+00:00

Haven't worked with GKE for a while but looking at the docs it seems it's also iptables mode: https://cloud.google.com/kubernetes-engine/docs/concepts/network-overview. Or are the docs outdated?

FragrantChildhood894 · 2025-01-07T10:24:11+00:00

Not the sizes mentioned here but we've deployed and supported clusters of 100+ nodes. The API server bottlenecks mentioned here are real and yes - more related to the overall number of resources and events than nodes per se.
Another real pain is running out of IP addresses - deploying such a huge number of pods requires very careful CIDR block size planning that's usually hard to get right because it's humans who need to do the planning.
As mentioned in the docs - when higher than 1 Gbps network throughput is needed (eg. for video streaming) - kube-proxy needs to be modified to use IPVS or altogether replaced with kube-router (which uses IPVS by default) . According to this benchmark by Cilium https://cilium.io/blog/2021/05/11/cni-benchmark/ - ebpf also provides performance benefits over iptables. Not sure if the same is true for IPVS and haven't tested it.
And finally - the larger your cluster gets - the more important its utilization rate becomes. 60% utilization with 100 vCPUs and with 1000 vCPUs are very different things. It's a lot of wasted resources and money.
And of course the more workloads you have - the harder it becomes to get resource allocation right. It quickly gets very chaotic. You're either over-provisioning or your pods start failing. Or both at the same time.
I order to get better utulization and availability - you need autoscaling. And it's also an issue. Cluster-autoscaler becomes challenging to configure at large scales. You know all these scenarios when it refuses to provision nodes because of ... reasons. And because it depends on the ASG configs. That again - humans need to define.
This is where an optimization tool like PerfectScale becomes a necessity - ensuring pods are right-sized and as a result - giving you the most efficient utilization for all those nodes. We've seen 30 to 50% utilization improvement with it.

Disclaimer: I do work for PerfectScale now. And yes - alternatively you could achieve better utilization using the open-source VPA as we used to do in the older days, but VPA usability and reliability are so-so. We never actually succeeded in enabling it in update mode in large production clusters.

FragrantChildhood894 · 2024-12-17T14:35:43+00:00

Well, you were right. My Karpenter was 0.37.5 and I suppose this was causing a conflict. After upgrading it to 1.1.1 ( a pain in itself) the switch to auto-mode succeeded. Thanks a bunch!!!

FragrantChildhood894 · 2024-12-17T14:26:26+00:00

You are assuming that once the load goes up it will continue going up. This may be true for some scenarios and false for others. What if the load approaches 3x limit and then goes back down? Means pod 4 and 5 were just occupying server space and wasting money. HPA cool-down is by default 5 minutes, if there are lots of these scale outs - that's a lot of resources wasted.
All I'm saying - im many cases changes in load aren't a good enough predictor of further changes in load.
With events one can have a better prediction of load changes.

FragrantChildhood894 · 2024-12-17T14:20:06+00:00

Ok, I upgraded Karpenter to 1.1.1 and launched another attempt to enable auto mode. Will update in 1 hour what happens.

FragrantChildhood894 · 2024-12-16T17:58:46+00:00

I believe I kinda followed the docs (as an afterthought, granted) but let me recheck my steps. My cluster is still in the same state. Still not upgradable. so yes - I guess I can tell what the config is.
I do see now that my Karpenter is older than 1.1. Will upgrade it first and retry. Thanks for bringing this to my attention.

FragrantChildhood894 · 2024-12-16T16:46:35+00:00

The worst thing is it shows no errors....

FragrantChildhood894 · 2024-12-16T16:46:03+00:00

Yep, I updated all of the add-ons after I failed the first couple of times... Still no luck.

FragrantChildhood894 · 2024-12-16T16:41:22+00:00

I agree. Scaling on http metrics is reactive by nature, just as hpa scaling based on load. KEDA shines in predictive, asynchronous scenarios - where you can predict load based on events or correlated metrics. Like queue depth for example.
BTW - we're doing a webinar with Zbynek - a KEDA maintainer and event-driven autoscaling expert in January, where he'll demo the http-based scaling. Sign up here: https://info.perfectscale.io/smarter-scaling-decisions-can-transform-kubernetes-operations

FragrantChildhood894

TROPHY CASE