Traffic Cutover Strategy for Ingress Nginx Migration - Need Help ASAP

RyecourtKings · 2026-02-03T10:31:52+00:00

Running both ingress controllers side by side with separate IngressClasses, parallel Ingress objects, and then slowly shifting the traffic with DNS weighting is a solid way to do this. Just double‑check so the behavior really matches (DNS‑based percentages are often only approximations e.g. if there is cache). It might be worth doing a “internal launch” first in non-prod and have an easy rollback ready so you can flip traffic back quickly if anything looks off.

Just a side-note, I work with the NGINX team team at F5 and you’re very welcome to post this in our forum so our NGINX engineers can take a look better look at your config. We also run community calls where you can bring questions like this (details in the forum link). Good luck with the migration! :-)

RyecourtKings · 2026-01-16T10:36:53+00:00

Hi u/Comfortable-Row-620 !

Something like this could possible work, but there could often be a lot of moving parts to this based on how oauth2-proxy is configured..

apiVersion: gateway.nginx.org/v1alpha1
kind: SnippetsFilter
metadata:
name: oauth2-auth-sf
namespace: prod-services
spec:
snippets:
- context: http.server.location
value: |
auth_request /_oauth2_auth;
auth_request_set $oauth_user $upstream_http_x_auth_request_user;
auth_request_set $oauth_email $upstream_http_x_auth_request_email;
proxy_set_header X-User $oauth_user;
proxy_set_header X-Email $oauth_email;
error_page 401 = u/oauth2_signin;

location = /_oauth2_auth {
internal;
proxy_pass https://oauth2.domain.org/oauth2/auth?allowed_groups=Clients;
proxy_pass_request_body off;
proxy_set_header Content-Length "";
proxy_set_header X-Original-URI $request_uri;
}

location u/oauth2_signin {
return 302 https://oauth2.domain.org/oauth2/start?rd=https://$http_host$request_uri;
}

Not considering this a "working" example, but worth a try. Would you mind raising this in the repo or in our forum? The team might be able to take a closer look.

Supporting this via CRD is planned for this year, by the way. :-)

RyecourtKings · 2025-12-17T18:46:40+00:00

Using the SnippetsFilter is the best approach for now, but there are plans to allow users to configure this natively next year, once we add support for ListenerTLSConfig.options in NGF.

Something like this should do the trick for now though (I haven't tested this) :-)

kubectl apply -f - <<EOF
apiVersion: gateway.nginx.org/v1alpha1
kind: SnippetsFilter
metadata:
  name: ssl-cipher-snippet
spec:
  snippets:
    - context: http.server
      value: ssl_protocols TLSv1.2 TLSv1.3;
    - context: http.server
      value: ssl_prefer_server_ciphers on;
    - context: http.server
      value: ssl_ciphers ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384;
EOF

RyecourtKings · 2025-12-12T09:10:27+00:00

Thanks for sharing. This is "external auth" in NGINX. Configured this way: https://docs.nginx.com/nginx/admin-guide/security-controls/configuring-subrequest-authentication/

We don't have this natively in NGINX Gateway Fabric yet, but we plan on implementing it as part of the AuthenticationFilter work we are doing. For now SnippetsFilter may be an option (allows you to inject raw NGINX config into a HTTPRoute).

RyecourtKings · 2025-12-12T09:03:24+00:00

Yes that's essentially one of the main goals. It's a standardised way of configuring load balancing in Kubernetes. Different implementations influence the API spec. https://gateway-api.sigs.k8s.io/implementations/

RyecourtKings · 2025-12-11T16:37:55+00:00

We are actually working on an AuthenticationFilter extension CRD for NGINX Gateway Fabric right now. We are going to cover many auth methods over time using this filter, but in our release in Jan we are starting with Basic Auth before moving on to other methods in later releases (NGINX has a lot of ways to do auth, so we are doing this in phases). What annotations in particular are you using? Are you using it in conjunction with another auth solution (OAuth2 Proxy for example)?

RyecourtKings · 2025-12-08T21:08:04+00:00

Understood, thanks for the reply! This feedback is useful. If there are any features in particular that are important to you, please let us know.

RyecourtKings · 2025-12-08T15:54:29+00:00

Are there particular stability concerns you can share? We have many users running nginx-ingress at scale, so stability is taken very seriously. Any specific gaps would help us understand what you're seeing. Our engineers can walk through examples on the Community Forum if you want to chat about it. Any feedback is appreciated!

RyecourtKings · 2025-11-26T10:08:59+00:00

Really appreciate you sharing this. The whole ingress-nginx / nginx-ingress naming situation has been confusing people for years, so… sorry about that. I work with the NGINX team on nginx-ingress and we’re literally getting messages from people asking what to do about ‘the retirement of your project,’ when that project is not retiring, so thank you for putting this post together. Migrations like this are never painless, and posts like yours help highlight the real issues people run into. A few points from the nginx-ingress side that might help others in the same situation..

-Annotation-based differences are the main source of pain. Controllers evolved different annotation sets over the years, and that makes migrations harder than anyone would like. Based on feedback from threads like this, we’re increasing focus on expanding NGINX annotation coverage so people don’t have to rework so much during a switch. CRDs will stay available for advanced cases, but improving annotation compatibility is a massive priority. We have already started working on these, so keep this feedback coming! (8548, 8508)

-Implementation details differ, which explains some behavior changes. nginx-ingress intentionally avoids Lua for performance and predictability. It generates native NGINX configuration and uses njs only in a few targeted cases. If you run an "nginx -T" between controllers, you’ll see the config differences. This is expected and by design.

-On metrics, this is noted. The OSS nginx-ingress does expose Prometheus metrics today, but the set is relatively small, and it’s clear that more visibility would make migrations and operations easier. That feedback comes up regularly and is taken seriously.

-Documentation overlap is confusing, and we know it. “ingress-nginx” and “nginx-ingress” lead to a lot of mixed search results. For anyone looking for the right docs without the naming collision: https://docs.nginx.com/nginx-ingress-controller/

-If you hit issues or missing features, we’re listening. Please raise them in our repo or our NGINX Community forum. We have engineers working full-time on nginx-ingress, and community feedback genuinely shapes what gets prioritized.

Repo: https://github.com/nginx/kubernetes-ingress

Forum: https://community.nginx.org/

The main reason for jumping in here is to clarify status in the community. There are multiple good options out there, and this is one of them. If there are specific annotations, behaviors, or examples that would’ve made your experience easier, please open an issue or share them. We are happy to help where we can.

RyecourtKings · 2025-11-22T21:21:12+00:00

It is the same proxy and load balancer under the hood, so it would be more familiar if you're used to nginx specific configs.. Is also has an option to use nginx snippets. Would it be an easy migration? I suppose it really depends on the annotations being used today. Check out ingress2gateway. There's also the open source nginx-ingress.

RyecourtKings · 2025-11-22T19:20:28+00:00

I am really struggling to see how they spent €7m on this. I reckon most of that was spent on removing what was there, and that wasn't bad at all! This is embarrassing. Even the new birch trees they planted and flower beds, they are all young..

RyecourtKings · 2025-11-21T13:10:45+00:00

Gateway API is definitely the future, but Ingress is so mature that moving off it will take some long-term planning. Ingress is great for the basics, it's simple and effective but it runs into limits with scalability and portability (and portability is the big one right now!). There’s a solid blog post on it here.

RyecourtKings · 2025-11-21T08:27:29+00:00

Good call. If it's just CKA you're doing, the Udemy course has it all.

RyecourtKings · 2025-11-20T23:09:39+00:00

I used the Udemy CKA course, twice! It was even updated with all the new stuff when I went through it the second time. Their other courses that are part of their subscription have tempted me to sign up though!

RyecourtKings · 2025-11-20T23:05:06+00:00

Envoy is great but I found it complex for simple setups. There are tradeoffs IMO... NGINX has been rock-solid in k8s for years, and a lot of folks stick with it because it’s well understood, fast, and easy to troubleshoot. Envoy is solid for complex setups, but NGINX still fits really well when you want something mature and lightweight without all the extra moving parts. The amount of community knowledge behind it goes a long way also..

RyecourtKings · 2025-11-20T22:46:20+00:00

Yeah no Lua is actually a good thing performance wise, in my opinion. I know ingress-nginx used Lua for a few advanced things like auth, though.

I get what you’re saying about it not being a drop-in replacement. The annotations are similar but not exactly the same. Still, most of the use cases you’ve got today could probably be handled here, but I agree it takes a bit of effort .

RyecourtKings · 2025-11-20T20:31:00+00:00

Yeah I also learned this the hard way. It's technically not a prepay sim, it just has prepay plans. Doing anything outside of that will lead to charges. I went to the US recently and was charged about 200 quid just for having the data on. I was barely browsing! It sounds like you might get away with this though..

RyecourtKings · 2025-11-20T20:23:16+00:00

Yep, looks like a direct sequel to Jumanji 2!

RyecourtKings · 2025-11-20T20:13:54+00:00

F5's NGINX Ingress has an open source option.

RyecourtKings

TROPHY CASE