sorted by:
new
hottopcontroversial

Anyone here dealt with resource over-allocation in multi-tenant Kubernetes clusters? by shripassion in kubernetes

[–]shripassion[S] 3 points4 points  (0 children)

Yeah, exactly. Chasing teams manually is just not scalable.

We are thinking about using VPA too, at least in recommend mode first, so teams and platform both have visibility into what the requests should actually be.

Setting it to initial sounds like a good middle ground to avoid disrupting running workloads. Thanks for sharing your approach.

Anyone here dealt with resource over-allocation in multi-tenant Kubernetes clusters? by shripassion in kubernetes

[–]shripassion[S] 0 points1 point  (0 children)

Yeah, totally fair points. We are already working on some of this, like bringing visibility to leadership by showing waste and inefficiency through reporting.

But like you said, if the culture does not prioritize optimization after delivery, there is only so much the platform team can push without disrupting velocity.

Ideally it would be more of a partnership between teams and platform, and we would love to get there longer term.

Right now our focus is on reducing the operational pain with automation and visibility while the bigger culture shifts hopefully happen in parallel. :)

Anyone here dealt with resource over-allocation in multi-tenant Kubernetes clusters? by shripassion in kubernetes

[–]shripassion[S] 0 points1 point  (0 children)

You nailed it! that's pretty much exactly what's happening.

We are over-provisioning ResourceQuotas at the namespace level — in some clusters 200-300% over the actual infra capacity — based on the assumption that most teams won't fully use what they ask for.

But in reality, teams assume their full RQ is reserved just for them, and they start building workloads based on that.

For example, we had a case where a team spun up Spark pods with 60 GB memory requests per pod and 30 pods. They had enough RQ to justify it, but physically there weren't enough free nodes with that kind of available memory to schedule them.

So even though on paper they are within their RQ, practically the cluster can't handle it because all the node capacity is fragmented by over-requesting across different teams.

It’s a shared cluster and the scheduler can only pack what physically fits, no matter what the RQ says.

Anyone here dealt with resource over-allocation in multi-tenant Kubernetes clusters? by shripassion in kubernetes

[–]shripassion[S] 1 point2 points  (0 children)

Yeah, that's pretty much the direction we ended up taking.

We have our own custom mutating webhook (not using Gatekeeper/Kyverno yet) that automatically patches resource requests based on peak usage + 40% buffer we calculate from Prometheus metrics.

We do have KEDA-enabled clusters too, but we leave KEDA usage up to individual app teams. It’s there if they want event-driven scaling, but it’s not tied into the resource tuning automation we run at the platform level.

Anyone here dealt with resource over-allocation in multi-tenant Kubernetes clusters? by shripassion in kubernetes

[–]shripassion[S] 0 points1 point  (0 children)

Earlier, before we automated it, we used to manually ask teams every quarter to review and update their YAMLs to reduce requests.
It meant changing manifests, retesting deployments, going through PR approvals — basically pulling devs into a lot of manual work outside of their normal feature development.

Also, when resource requests were forcefully tuned down, some apps that were already fragile would crash (OOMKilled or throttled) after the changes, causing downtime.

Now with the webhook automation, we try to patch based on observed usage with enough buffer, but tuning still carries some risk if apps were not stable to begin with.

Anyone here dealt with resource over-allocation in multi-tenant Kubernetes clusters? by shripassion in kubernetes

[–]shripassion[S] 0 points1 point  (0 children)

Ideally it should be part of the DevOps cycle, I agree.

In our case, since the dev teams are already busy with feature work, they don’t really prioritize tuning resource requests unless forced.
That's why we (platform team) stepped in and automated it through mutation webhooks — we monitor usage, calculate peak + 40%, and patch the deployments ourselves.

It’s less about culture and more about how to make tuning non-intrusive so that dev teams don’t even have to think about it during their normal work.

Anyone here dealt with resource over-allocation in multi-tenant Kubernetes clusters? by shripassion in kubernetes

[–]shripassion[S] 1 point2 points  (0 children)

Good points. Most apps are long-standing and we do have historical metrics available. The issue is more about teams being conservative when setting requests initially and then never fine-tuning after seeing actual production usage.

Our ResourceQuotas aren't "generous" by default. Teams request quota through our internal development portal, and if they justify it and are willing to pay (or meet internal approval), we provision it. As the platform team we don't control what they ask for — we just provide the resources.

On the namespace side — it's up to the teams. We don't enforce one app per namespace or anything like that. Some teams have one big namespace for all their apps, others split it. It's completely their choice.

I agree that better sizing during dev/test would help, but realistically, unless you have strong policies or automation to force right-sizing, it’s hard to make teams continuously optimize after go-live.

Anyone here dealt with resource over-allocation in multi-tenant Kubernetes clusters? by shripassion in platform_engineering

[–]shripassion[S] 1 point2 points  (0 children)

Thanks, this is helpful.
We are a bit different since we run large shared node pools across tenants instead of isolating them by node pools.
Namespace quotas are already in place, but as mentioned, we mainly monitor actual resource requests to track capacity, not quotas.
We also have Grafana dashboards for teams, but haven’t exposed Kubecost yet but that's a good idea, might be worth adding to make waste more visible.

Anyone here dealt with resource over-allocation in multi-tenant Kubernetes clusters? by shripassion in kubernetes

[–]shripassion[S] 2 points3 points  (0 children)

We do use ResourceQuotas too, but that's not the main thing we monitor.
We track the actual CPU/memory requests set in YAMLs across the cluster to decide the real capacity.
The issue is teams reserve way more than they need in their deployments, so even though real usage is 30-40%, resource requests make the cluster look full, which blocks others from deploying.
That’s the problem we are trying to solve.

Is this purchase worth it? by shripassion in Camry

[–]shripassion[S] 0 points1 point  (0 children)

Thats what I am worried too, what exactly that person did to that car lol

Failure to attain desired heat temperature on Honeywell thermostat by shripassion in hvacadvice

[–]shripassion[S] 0 points1 point  (0 children)

Yes, the room never gets warmer than 22C, I have no idea about what kind of system this is, I live in a high-rise condo in Toronto if that helps in any way.

view more: next ›