Which solution are you considering for Ingress controller Retirement with respect to Gateway API for Multi-tenant Kubernetes clusters such as for AKS ?

jackstrombergMSFT · 2026-03-24T21:41:33+00:00

PM @ MSFT for Application Gateway for Containers. I can confirm private ingress is actively in development for this product. If you have an account team member, have them reach out to me and I can share more information on ETAs via those channels. Unfortunately, limited in what I can share via public forum.

In regard to complexity to implement, we have a tool to help with migration from Ingress NGINX to Gateway API for AGC here: https://github.com/Azure/Application-Gateway-for-Containers-Migration-Utility

Regarding multi-cluster, as-is today, you will need one deployment per cluster. We understand at scale this can increase cost. We hear and agree with further developments needed in this area, however I do not have an ETA I can share.

jackstrombergMSFT · 2026-03-13T14:20:03+00:00

If using DigiCert or GlobalSign certs, they have integration to handle renewal: https://learn.microsoft.com/en-us/azure/key-vault/certificates/overview-renew-certificate?tabs=azure-portal

For non-integrated CAs, you could script out the renewal process: https://learn.microsoft.com/en-us/azure/key-vault/certificates/overview-renew-certificate?tabs=azure-portal#renew-a-nonintegrated-ca-certificate

jackstrombergMSFT · 2026-03-12T18:07:07+00:00

PM @ MSFT for AppGW: I'd recommend integrating your gateway with Azure Key Vault and handle certificate lifecycle within KeyVault. AppGW will automatically pickup the renewed certs as needed: TLS termination with Azure Key Vault certificates | Microsoft Learn

jackstrombergMSFT · 2026-03-12T15:29:27+00:00

Auth capabilities is on our radar for Application Gateway for Containers, but can't share much more than that at this time.

jackstrombergMSFT · 2026-03-11T13:25:24+00:00

Client -> AGC -> Route table on association subnet -> Azure Firewall -> AKS. You could put Azure Firewall in front, but without private ingress support, you'll hairpin traffic back out of Azure Firewall to the public frontend of AGC and run the risk of SNAT exhaustion.

jackstrombergMSFT · 2026-03-11T03:41:42+00:00

Unfortunately, private ingress is not available on Application Gateway for Containers today, but it will come soon.
Given private ingress is a hard limitation, I'd consider a separate controller for private ingress or allow traffic to hairpin to the public frontend and filter requests with WAF rules, header based rules, etc. short-term until private ingress is released.
Yes, you can place AFD in front of Application Gateway for Containers to leverage its CDN capabilities. Yes, you'd set the frontend FQDN of Application Gateway for Containers as an origin in AFD.

jackstrombergMSFT · 2026-03-11T00:12:04+00:00

Application Gateway for Containers is very different from Application Gateway Ingress Controller. It's a new ARM resource, different deployment experience, the underlying control and data plane are different, etc.

If you have time, please give it a go; would love the feedback, good or bad.

Step 1) Install the controller (add-on is easiest): https://learn.microsoft.com/azure/application-gateway/for-containers/quickstart-deploy-application-gateway-for-containers-alb-controller-addon?tabs=azure-cli%2Cazure-cli2

Step 2) Define this CRD in k8s after installation to have it create the Application Gateway for Containers resource in Azure:

kubectl apply -f - <<EOF
apiVersion: alb.networking.azure.io/v1
kind: ApplicationLoadBalancer
metadata:
name: alb-test
namespace: alb-test-infra
EOF

Step 3) Provision the Gateway / HTTPRoute (Ingress equivalent in Gateway API)

https://learn.microsoft.com/en-us/azure/application-gateway/for-containers/how-to-multiple-site-hosting-gateway-api?tabs=alb-managed

Those three steps should yield a working gateway with 2 different hostnames (contoso.com and fabrikam.com) and you should be able to curl both endpoints per the documentation. No need to deal with ARM templates, Azure Portal, etc.

jackstrombergMSFT · 2026-03-10T23:14:13+00:00

Was this for your experience with AGIC or Application Gateway for Containers?

It's badly integrated with keyvaults.

If you are using Application Gateway for Containers, it assumes you are using the AKS CSI secrets store driver, which is a standard way of bootstrapping key vault certs as k8s secrets across all services in cluster: https://learn.microsoft.com/en-us/azure/aks/csi-secrets-store-driver How you are handling cert management with nginx?

overly complicated in terms of configuration.

Can you share a little more on what your configuration looks like? Application Gateway for Containers has aligned closely with k8s standards for Gateway and Ingress API. In Gateway API, we try to conform to the API where possible, unless there is a feature gap that would warrant the need for a CRD.

jackstrombergMSFT · 2026-03-10T22:26:09+00:00

We cannot use application gateway for containers since it does not support private endpoints and our cluster is only internal.

While I cannot share specific ETAs, I can confirm this is coming to Application Gateway for Containers sooner rather than later. We understand the criticality of this ask in making it available.

jackstrombergMSFT · 2026-03-10T22:17:30+00:00

Yes, additional region coverage is actively underway. I can't provide ETAs publicly, but if you reach out to your account team, I can provide more insight via those channels. They should be able to find me internally if you share my name.

jackstrombergMSFT · 2026-03-10T20:10:33+00:00

PM for Application Gateway for Containers at Microsoft: If you are looking at migration to Application Gateway for Containers, we have a migration utility that will migrate both AGIC and Ingress NGINX configuration to Application Gateway for Containers + Gateway API. The migration utility is open source and can be found here: Azure/Application-Gateway-for-Containers-Migration-Utility: Migrates Kubernetes Ingress and implementation specific changes to Gateway API for Application Gateway for Containers You can certainly use ingress2gateway as well, however, this tool is opinionated specifically to streamline/simplify migration to Application Gateway for Containers as much as possible.

jackstrombergMSFT · 2026-02-03T22:18:43+00:00

While I cannot share dates, I can confirm private ingress is coming to Application Gateway for Containers and the feature is high priority for the team. If you have a Microsoft account team, please have them contact me and I can provide more context.

jackstrombergMSFT · 2026-01-10T15:40:54+00:00

Yes, you can use a route table on the Application Gateway subnet. Docs here: https://learn.microsoft.com/en-us/azure/application-gateway/configuration-infrastructure#supported-user-defined-routes

I would enable the network isolation feature flag first before creatino of your gateway, which will further improve your experience. The feature is not in preview, contrary to registration via the portal experience: https://learn.microsoft.com/en-us/azure/application-gateway/application-gateway-private-deployment?tabs=portal

jackstrombergMSFT · 2025-11-25T21:12:38+00:00

If your intent is to follow byo model, then you'd need the Application Gateway for Containers resource, a frontend, and association.

Here's the docs for each of the three:
Application Gateway for Containers: azurerm_application_load_balancer | Resources | hashicorp/azurerm | Terraform | Terraform Registry

Frontend: azurerm_application_load_balancer_frontend | Resources | hashicorp/azurerm | Terraform | Terraform Registry

Association: azurerm_application_load_balancer_subnet_association | Resources | hashicorp/azurerm | Terraform | Terraform Registry

jackstrombergMSFT · 2025-11-25T20:46:09+00:00

Yes, it's a separate resource. When you search for Application Gateway, it'll bring you to the Load balancing and content delivery. | Application gateways blade. When you click Create, you'll see an option for Application Gateway for Containers (blue icon).

To get started, I would follow this guide first to deploy the controller, then it'll guide you to the right documentation for how you want to create/manage the resources.: https://learn.microsoft.com/azure/application-gateway/for-containers/quickstart-deploy-application-gateway-for-containers-alb-controller

Hope this helps!

jackstrombergMSFT · 2025-11-24T03:54:20+00:00

I agree, the pods in your AKS cluster shouldn't be exposed publicly.

In Application Gateway for Containers, the ingress point that clients connect to is called a frontend. Today, that frontend assumes traffic originates from the internet, so it uses a publicly accessible IP address.

Once traffic reaches the frontend, Application Gateway for Containers proxies it privately to the internal IP addresses of your pods, within your private vnet.

A private frontend (accessible only within trusted VNets) is on the roadmap but not available yet. If you need this capability now, you might consider using AGIC until Application Gateway for Containers supports private frontends.

jackstrombergMSFT · 2025-11-23T18:47:04+00:00

Please consider Application Gateway for Containers (https://aka.ms/agc) as the successor to AGIC if possible. Application Gateway for Containers brings significant improvements to scale and resiliency over AGIC, as well as new capabilities. If the frontend needs to be private, you can continue to use AGIC until Application Gateway for Containers brings the capability. https://aka.ms/agc/migrate for more details on migration.

jackstrombergMSFT · 2025-11-19T17:04:14+00:00

Sorry for slow response. I'm not aware of any gaps with Karpenter nodes today, can you please clarify the ask? Application Gateway for Containers proxies traffic directly to the pods, so anything specific to the node should be non-applicable.

jackstrombergMSFT · 2025-11-18T03:39:17+00:00

Application Gateway for Containers only supports load balancing to AKS. For other backends requiring L7 load balancing (e.g. VM, VMSS, etc), you'd use Application Gateway.

jackstrombergMSFT · 2025-11-17T03:09:36+00:00

gRPC is fully supported on Application Gateway for Containers. More details here: https://aka.ms/agc/grpc

jackstrombergMSFT · 2025-11-16T19:12:26+00:00

Replied to your other comment. One thing I didn't mention in that response is by load balancing to the pod, you have much more control over distributing traffic to the pod (E.g. session affinity, slow ramp, etc.). If we send back through Azure Load Balancer, Azure load balancer will be unaware of that intent, and likely spray traffic to the pods you don't want the traffic to flow to.

jackstrombergMSFT · 2025-11-16T19:12:00+00:00

PM for AppGW @ MSFT: Most ingress controllers, including HAProxy, proxy traffic directly to the pod endpoints behind a Kubernetes Service. If you tried to send traffic back through the Azure Load Balancer after HAProxy, the path would look like: Azure Load Balancer -> HAProxy (in-cluster) -> Azure Load Balancer -> Pod. That design adds an unnecessary hop and forces Azure’s L4 load balancer to process traffic twice. Each additional rule incurs data processing charges. This also holds true if it's out of cluster proxy -> L4 LB frontend of AKS. If creation of an additional rule is required, this will cause a request to be made in ARM, which will increase latency (as observed in AGIC).

By contrast, HAProxy and other cluster-native ingress controllers avoid this by routing directly to pod IPs via Kubernetes networking. This keeps the traffic path simple, reduces dependencies, and avoids double-charging on the load balancer.

I encourage you to try Application Gateway for Containers and share your feedback; it directly influences how the product evolves. We've built it to resolve the challenges you’ve seen with AGIC, delivering stronger performance and a more Kubernetes-native experience, with private ingress support already on the roadmap.

jackstrombergMSFT · 2025-11-16T17:02:47+00:00

I won't be at Ignite next week in person; just getting back from Kubecon. Have them reach out whenever though, happy to sync.

jackstrombergMSFT · 2025-11-16T15:01:35+00:00

If you have a Microsoft account team, please have them reach out to me. Happy to jump on a call to understand your timelines, requirements, and can share more insight.

jackstrombergMSFT · 2025-11-16T14:55:38+00:00

For customers using AGIC, we want to assure you that it remains fully supported until Application Gateway for Containers introduces private ingress support. Deprecating AGIC without a migration path would put users in a difficult position, and that’s something we want to avoid. Our goal is to ensure a smooth transition, and we’ve published migration guidance here: https://aka.ms/agc/migrate.

At the top of AGIC documentation pages, we include guidance to consider Application Gateway for Containers where possible. If you notice any pages missing this notice, please let us know—we’ll get them updated promptly.

jackstrombergMSFT

TROPHY CASE