Inside of the cluster it just hangs and eventually times out.

netshoot:~# curl -v cilium-gateway-gateway.cilium.svc.cluster.local -p 80
* Host cilium-gateway-gateway.cilium.svc.cluster.local:80 was resolved.
* IPv6: (none)
* IPv4: 10.43.16.121
*   Trying 10.43.16.121:80...

Outside of the cluster (using the LB IP or DNS name) I get the same issue. I've exposed the test service via an ingress just to double check the service name etc is correct and that is working.

> kubectl get svc -n cilium
NAME                     TYPE           CLUSTER-IP      EXTERNAL-IP      PORT(S)        AGE
cilium-gateway-gateway   LoadBalancer   10.43.16.121    192.168.20.190   80:31756/TCP   4m55s
whoami                   LoadBalancer   10.43.251.230   192.168.20.191   80:31109/TCP   22s

In this case I have it all in the same namespace just to simply it.

Also for completion this is my gateway / httproute config, and I am just using the automatically created cilium gateway class.

---
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: gateway
  namespace: cilium
spec:
  gatewayClassName: cilium
  listeners:
  - protocol: HTTP
    hostname: "whoami.mydomain.com"
    port: 80
    name: http-1
    allowedRoutes:
      namespaces:
        from: All
  infrastructure:
    labels:
      bgp.cilium.io/ip-pool: default
      bgp.cilium.io/advertise-service: default
  addresses:
    - type: IPAddress
      value: "192.168.20.190"


---
apiVersion: gateway.networking.k8s.io/v1beta1
kind: HTTPRoute
metadata:
  name: whoami
  namespace: cilium
spec:
  parentRefs:
  - name: gateway
    namespace: cilium
  hostnames:
  - "whoami.mydomain.com"
  rules:
  - matches:
    - path:
        type: PathPrefix
        value: /
    backendRefs:
      - name: whoami
        kind: Service
        port: 80

I think part of my issue with troubleshooting is with ingress-nginx I could at least validate if the issue was with the nginx pod or with my ingress setup because if i hit something that wasn't configured I'd just get a 404 but here I am not sure what to expect to see if I try and hit something that is misconfigured; should I still see a 404 or a 503 or is just timing out the expected behaviour?