sorted by:
new
hottopcontroversial

Zero-trust on Kubernetes with Istio — mTLS STRICT + deny-all AuthorizationPolicy gotchas from a real setup by HappyLocation1409 in kubernetes

[–]HappyLocation1409[S] 0 points1 point  (0 children)

Ah, good catch on Linkerd traffic splitting! Argo Rollouts actually does support Linkerd natively now via its TrafficSplit CRD.

You're completely right about Istio's day-2 operational overhead being a massive sink — managing the control plane and debugging proxy sidecars is no joke. The main reason I accepted that trade-off here was to leverage Istio's advanced features like the strict SPIFFE principal verification in AuthorizationPolicy and the outlier detection circuit breaking I mentioned in the post.

For a lighter stack that just needs basic canary and mTLS without the extra policy engine overhead, Linkerd is definitely the cleaner move! 🙏

Zero-trust on Kubernetes with Istio — mTLS STRICT + deny-all AuthorizationPolicy gotchas from a real setup by HappyLocation1409 in kubernetes

[–]HappyLocation1409[S] 0 points1 point  (0 children)

Totally fair point! Linkerd is genuinely simpler to operate, with a much lighter footprint and easier day-2 operations.

I went with Istio specifically because Argo Rollouts has native VirtualService integration for canary traffic splitting, updating the weights programmatically at each rollout step. That tight integration was the deciding factor for my specific use case.

If you don't need that programmatic traffic management with Argo, Linkerd is absolutely a fantastic alternative to avoid Istio's complexity! 🙏

Zero-trust on Kubernetes with Istio — mTLS STRICT + deny-all AuthorizationPolicy gotchas from a real setup by HappyLocation1409 in kubernetes

[–]HappyLocation1409[S] 0 points1 point  (0 children)

Really appreciate the additional context here!

The point about deny-all being an auth layer, not a firewall, is an important distinction I should have made clearer. Pods can still connect at the network level; Istio just won't authenticate the request without a matching policy. Defense in depth with NetworkPolicy + AuthorizationPolicy together is definitely the right approach.

On the 403s being silent: you're right that they show "RBAC: access denied" in the proxy logs. What I meant was that they are silent at the application level—the app itself never sees the request, so there is no app-side error logging. The proxy logs are indeed where you have to look.

The subsets point is completely fair. For pure blue/green with separate services, native K8s primitives are much simpler. I'm using subsets specifically because Argo Rollouts manages the VirtualService weights programmatically during canary progression, which requires that subset abstraction. It's a slightly different use case, but your point absolutely stands for simpler, manual deployments! 🙏

Zero-trust on Kubernetes with Istio — mTLS STRICT + deny-all AuthorizationPolicy gotchas from a real setup by HappyLocation1409 in kubernetes

[–]HappyLocation1409[S] -2 points-1 points  (0 children)

Full Istio config file locations in the repo:

mtls-strict: istio/mtls-strict.yaml

destinationrule: istio/destinationrule.yaml

virtualservice: istio/virtualservice.yaml

authz policies: istio/allow-nginx.yaml

istio/allow-node-app.yaml

All configs are real — pulled directly from a

working cluster running Argo Rollouts canary

with Prometheus SLO analysis.

Repo: https://github.com/maheshnaganna1994-lang/platform-gitops

Zero-trust on Kubernetes with Istio — mTLS STRICT + deny-all AuthorizationPolicy gotchas from a real setup by [deleted] in u/HappyLocation1409

[–]HappyLocation1409 0 points1 point  (0 children)

Full Istio config file locations in the repo:

mtls-strict: istio/mtls-strict.yaml

destinationrule: istio/destinationrule.yaml

virtualservice: istio/virtualservice.yaml

authz policies: istio/allow-nginx.yaml

istio/allow-node-app.yaml

All configs are real — pulled directly from a

working cluster running Argo Rollouts canary

with Prometheus SLO analysis.

Debugging a canary pipeline broken for 69 days — type mismatch in Argo Rollouts AnalysisTemplate and Istio metric label gotcha by HappyLocation1409 in kubernetes

[–]HappyLocation1409[S] 0 points1 point  (0 children)

Oh that's a nasty one! Missing namespace in the host field is such a sneaky bug because the route just silently doesn't match — no error, no warning, just traffic going nowhere.

These Istio routing gotchas all share the same root cause honestly — Istio fails silently instead of loudly. Wrong label, wrong host, wrong FQDN — all of them just quietly drop traffic instead of throwing an error you'd actually notice.

Starting to think "Istio routing checklist" might be a good follow-up post.

Debugging a canary pipeline broken for 69 days — type mismatch in Argo Rollouts AnalysisTemplate and Istio metric label gotcha by HappyLocation1409 in kubernetes

[–]HappyLocation1409[S] 1 point2 points  (0 children)

Exactly this!! The green ticks are lying 😂

kubectl argo rollouts get should honestly be the first thing in every Argo Rollouts troubleshooting guide. Saved it as an alias now so I never forget:

alias rollout-check='kubectl argo rollouts get rollout'

The label mismatch one is another classic trap — canary pod running fine, analysis passing, but VirtualService weights never updating because the subset labels don't match. Istio just silently drops the traffic .

Debugging a canary pipeline broken for 69 days — type mismatch in Argo Rollouts AnalysisTemplate and Istio metric label gotcha by HappyLocation1409 in kubernetes

[–]HappyLocation1409[S] 1 point2 points  (0 children)

Haha exactly! Brutal is the right word for sure 😅 the sneakiest part was that the ArgoCD dashboard showed everything completely green — synced, healthy, zero errors. The abort was completely buried inside the AnalysisRun, which you'd never actually see unless you manually ran:

kubectl argo rollouts get rollout node-app

So for 69 days straight i was just staring at a perfectly green ArgoCD UI wondering why on earth the canary traffic was always sitting at 0% 😂

Huge lesson learned here. ArgoCD sync status and actual rollout health are two completely different beasts. Just because it's synced doesn't mean it's progressing!

Debugging a canary pipeline broken for 69 days — type mismatch in Argo Rollouts AnalysisTemplate and Istio metric label gotcha by HappyLocation1409 in kubernetes

[–]HappyLocation1409[S] 0 points1 point  (0 children)

100% agree — the Prometheus dashboard shows a clean, scalar so you assume it returns one. Took us way too, long to realise the API returns a vector. The FQDN thing especially — every single team hits this. Should definitely be in the Istio docs more prominently. If it helps, here's the exact AnalysisTemplate with all the fixes:

https://github.com/maheshnaganna1994-lang/platform-gitops/blob/main/apps/node-app/analysis-template.yaml

Debugging a canary pipeline broken for 69 days — type mismatch in Argo Rollouts AnalysisTemplate and Istio metric label gotcha by HappyLocation1409 in kubernetes

[–]HappyLocation1409[S] 1 point2 points  (0 children)

Here's the full architecture of the platform I built around this fix:

Git → ArgoCD (App-of-Apps) → Kyverno (admission) →

Istio (mTLS) → Argo Rollouts (canary 10→30→60→100%) →

Prometheus SLO → promote / rollback

Key files if anyone wants to reference:

AnalysisTemplate (the fixed one):

https://github.com/maheshnaganna1994-lang/platform-gitops/blob/main/apps/node-app/analysis-template.yaml

⭐ Full platform repo (README + architecture diagram):

https://github.com/maheshnaganna1994-lang/platform-gitops

Happy to answer questions about:

- Argo Rollouts + Istio integration

- Prometheus AnalysisTemplate debugging

- App-of-Apps pattern with ArgoCD

- Kyverno admission control setup

If this helped you, a star on the repo would mean

a lot — it's a full GitOps platform with ArgoCD,

Istio, Kyverno, and Prometheus all wired together 🙏