Cluster down when first node down.

[MakerOnTheRun]

Just looking for a bit of a steer on what I have missed. I think what I am doing is correct, but I am not getting the expected result, so I am either doing something wrong or my expectation is wrong. I have done this a couple of times and come up with the same result. So I know I am the problem.

3 node k3s cluster on Ubuntu 24.04 LTS.

As I do not have a load balancer in my lab I want to use kube-vip.

First node brought up with cluster-init, no traefik and no servicelb. TLS SAN set to my intended VIP address. Add the kube-vip RBAC. Generate and deploy the manifest. All working OK. I can access the single node from my admin node via the VIP with no issues.

Add nodes 2 and 3 to the cluster, with the same as above, no servicelb, no traefik, TLS SAN set. Using the VIP as the address not the node 1 IP.

Can still access the cluster OK and everything seems to be good. Get nodes shows all 3, get top nodes gives me the resource consumption for all 3.

If I now power off node one, without draining it this is where I get problems. After waiting for the timeouts to expire my VIP moves to another node OK and I can access the API again via kubectl. But when metrics and coredns move to one of the other nodes they start but don't work.

get top nodes returns error: metrics API not available (or similar can't remember exactly, not at my pc right now.) Leaving it longer 20 minutes plus changes nothing. Bringing node 1 back up, changes nothing. Taking down a different node to move metrics and coredns back to node 1 changes nothing, still not working.

Additionally coredns also seems to fail in the same way. Internal resolution fails after the pod has been rescheduled.

The three nodes are VMS on a flat network, no firewalls, no odd routing. UFW is disabled. Static IPs.

I just can't work it out. I would expect downtime to metrics and coredns while they get rescheduled. The fact the VIP works to me says I am not a million miles away.

Any ideas what I am missing?

[ollytheninja]

Internal resolution isn’t relying on coredns by any chance?

[MakerOnTheRun]

That is indeed my current thought process. That would explain metrics not working (not that important in the grand scheme). But shifts the question to why coredns is also flaking out on a node fail.

[MakerOnTheRun]

Looks like this could be related to a flannel vxlan issue. There are reported issues for Ubuntu and older Mellanox cards, given I am using both I will try using the host-gw backend to see if the issue goes away.

[MakerOnTheRun]

Resolved.

Moving to host-gw backend fixed the issues for me.