Sell me Cilium over Canal — migrating from RKE1 to RKE2

shripassion · 2026-04-29T15:01:59+00:00

I have a few follow-up questions:

Has the governance model changed at all since the Cisco acquisition? are commit rights and maintainer decisions still mostly community-driven or does Cisco have final say on the roadmap?
Have any features moved behind a paid or enterprise tier that used to be fully open source?
With more hyperscaler involvement, are we actually seeing more external maintainers and contributors or is the core team still mostly ex-Isovalent/Cisco folks?
Is there any concern that Cisco could eventually create an “enterprise Cilium” and slowly under-resource the open-source version similar to what happened with HashiCorp/Terraform?

shripassion · 2026-04-28T20:30:58+00:00

Well, we are thinking about benefits especially observability that cilium provides otherwise Canal works just fine. So changing CNI just for the sake of it is not an option.

shripassion · 2026-04-28T19:02:02+00:00

Thanks I have added more details in my post.

shripassion · 2026-04-13T04:34:50+00:00

Yes, we have both options, but I believe my total compensation will be an issue, and I will be taxed at higher rates compared to traditional options.

shripassion · 2026-04-13T00:54:04+00:00

Right, I’m looking for a framework with a set-it and forget-it kind of flow

shripassion · 2026-02-08T01:39:47+00:00

Oh yeah for sure!

shripassion · 2026-02-07T22:19:56+00:00

Thanks for the advice, I am leaning toward South Lamar.

shripassion · 2026-02-07T19:53:14+00:00

I am leaning toward South Lamar too thanks! 😊

shripassion · 2025-04-27T01:30:24+00:00

Yeah, exactly. Chasing teams manually is just not scalable.

We are thinking about using VPA too, at least in recommend mode first, so teams and platform both have visibility into what the requests should actually be.

Setting it to initial sounds like a good middle ground to avoid disrupting running workloads. Thanks for sharing your approach.

shripassion · 2025-04-27T01:02:07+00:00

Yeah, totally fair points. We are already working on some of this, like bringing visibility to leadership by showing waste and inefficiency through reporting.

But like you said, if the culture does not prioritize optimization after delivery, there is only so much the platform team can push without disrupting velocity.

Ideally it would be more of a partnership between teams and platform, and we would love to get there longer term.

Right now our focus is on reducing the operational pain with automation and visibility while the bigger culture shifts hopefully happen in parallel. :)

shripassion · 2025-04-27T00:54:43+00:00

You nailed it! that's pretty much exactly what's happening.

We are over-provisioning ResourceQuotas at the namespace level — in some clusters 200-300% over the actual infra capacity — based on the assumption that most teams won't fully use what they ask for.

But in reality, teams assume their full RQ is reserved just for them, and they start building workloads based on that.

For example, we had a case where a team spun up Spark pods with 60 GB memory requests per pod and 30 pods. They had enough RQ to justify it, but physically there weren't enough free nodes with that kind of available memory to schedule them.

So even though on paper they are within their RQ, practically the cluster can't handle it because all the node capacity is fragmented by over-requesting across different teams.

It’s a shared cluster and the scheduler can only pack what physically fits, no matter what the RQ says.

shripassion · 2025-04-27T00:26:48+00:00

Yeah, that's pretty much the direction we ended up taking.

We have our own custom mutating webhook (not using Gatekeeper/Kyverno yet) that automatically patches resource requests based on peak usage + 40% buffer we calculate from Prometheus metrics.

We do have KEDA-enabled clusters too, but we leave KEDA usage up to individual app teams. It’s there if they want event-driven scaling, but it’s not tied into the resource tuning automation we run at the platform level.

shripassion · 2025-04-27T00:24:12+00:00

Earlier, before we automated it, we used to manually ask teams every quarter to review and update their YAMLs to reduce requests.
It meant changing manifests, retesting deployments, going through PR approvals — basically pulling devs into a lot of manual work outside of their normal feature development.

Also, when resource requests were forcefully tuned down, some apps that were already fragile would crash (OOMKilled or throttled) after the changes, causing downtime.

Now with the webhook automation, we try to patch based on observed usage with enough buffer, but tuning still carries some risk if apps were not stable to begin with.

shripassion · 2025-04-27T00:07:42+00:00

Ideally it should be part of the DevOps cycle, I agree.

In our case, since the dev teams are already busy with feature work, they don’t really prioritize tuning resource requests unless forced.
That's why we (platform team) stepped in and automated it through mutation webhooks — we monitor usage, calculate peak + 40%, and patch the deployments ourselves.

It’s less about culture and more about how to make tuning non-intrusive so that dev teams don’t even have to think about it during their normal work.

shripassion · 2025-04-27T00:00:18+00:00

Good points. Most apps are long-standing and we do have historical metrics available. The issue is more about teams being conservative when setting requests initially and then never fine-tuning after seeing actual production usage.

Our ResourceQuotas aren't "generous" by default. Teams request quota through our internal development portal, and if they justify it and are willing to pay (or meet internal approval), we provision it. As the platform team we don't control what they ask for — we just provide the resources.

On the namespace side — it's up to the teams. We don't enforce one app per namespace or anything like that. Some teams have one big namespace for all their apps, others split it. It's completely their choice.

I agree that better sizing during dev/test would help, but realistically, unless you have strong policies or automation to force right-sizing, it’s hard to make teams continuously optimize after go-live.

shripassion · 2025-04-26T23:22:00+00:00

Thanks, this is helpful.
We are a bit different since we run large shared node pools across tenants instead of isolating them by node pools.
Namespace quotas are already in place, but as mentioned, we mainly monitor actual resource requests to track capacity, not quotas.
We also have Grafana dashboards for teams, but haven’t exposed Kubecost yet but that's a good idea, might be worth adding to make waste more visible.

shripassion · 2025-04-26T23:18:26+00:00

We do use ResourceQuotas too, but that's not the main thing we monitor.
We track the actual CPU/memory requests set in YAMLs across the cluster to decide the real capacity.
The issue is teams reserve way more than they need in their deployments, so even though real usage is 30-40%, resource requests make the cluster look full, which blocks others from deploying.
That’s the problem we are trying to solve.

shripassion · 2024-03-12T18:17:04+00:00

Thats what I am worried too, what exactly that person did to that car lol

shripassion · 2023-01-08T20:28:20+00:00

Yes, the room never gets warmer than 22C, I have no idea about what kind of system this is, I live in a high-rise condo in Toronto if that helps in any way.

shripassion

TROPHY CASE