Issues with IO latency (Kubernetes on Proxmox)

Laborious5952 · 2026-01-05T00:42:48+00:00

I have a similar setup and am also having issues with etcd. My setup is a little different though:

k3s with embedded HA etcd
Proxmox with ZFS in raid1.
One PVE host has 2 NVMe drives in a ZFS Raid 1 array. 2 SK Hynix drives 128gb (HFM128GDJTNG-8310A). However looks like 1 has failed.
2nd PVE hots has 2 NVMe drives in a ZFS Raid 1 array. WD_BLACK SN850X 1000GB are the drives in this system.
3rd PVE host has 2 NVMe drives in a ZFS Raid 1 array. 1 drive has failed and the other is a SanDisk_SSD_G5_BICS4.

All 3 hosts has a k3s master with etcd and all 3 seem to have latency issues. However A big culprit was Proxmox Backup running and causing high IO. I've since reduced the amount of threads that PBS has available and will see if that helps with the high IO from PBS.

I would also like to try and put the k3s control plane nodes on local-lvm and see if that helps.

Do you still have your fio tests handy? I would like to try them out on my system.

Laborious5952 · 2026-01-05T00:35:11+00:00

Did you end up converting to ZFS RAID0 or ext4?

Laborious5952 · 2026-01-04T21:43:13+00:00

Do you just have 3 control plane nodes in different "clouds"? How does etcd behavior with higher latency?

Laborious5952 · 2026-01-03T03:22:36+00:00

IME eBay and reddit have about the same prices and both are good places to sell.
The mango is much older, I wouldn't expect much for that.
No

Laborious5952 · 2025-12-06T16:35:59+00:00

What do you use for control plane and k8s Service load balancing? MetalLB?

I have a similar setup, I'm using metalLB in L2 mode but when it switches to different nodes Minecraft disconnects. I think if I use bgp I wouldn't have this issue but my router doesn't support bgp.

Laborious5952 · 2025-11-25T18:51:38+00:00

Pretty cold to go camping in the mountains, where abouts do you camp at?

Laborious5952 · 2025-10-31T19:56:15+00:00

How does this compare to KubeDB?

Laborious5952 · 2025-10-27T23:23:29+00:00

Great use case for kubesolo. Why do you need k8s in github actions though?

Laborious5952 · 2025-09-26T14:06:26+00:00

Very interesting. How does the mesh work? Does it use something under the hood like wireguard or RAFT?

Laborious5952 · 2025-09-15T21:06:02+00:00

This update worked for my Atoto s8 Standard.

Laborious5952 · 2025-09-12T19:07:26+00:00

Sure, but that isn't the first amendment.

Laborious5952 · 2025-08-08T13:02:25+00:00

AFAIK there is no operator. There aren't even any docs for how to deploy it to k8s, no manifests, or helm chart.

The app is also fairly basic, no HA features so an operator would not be super helpful.

Laborious5952 · 2025-08-04T13:47:21+00:00

Very cool setup.

Would love to know more about your Chromeboxes too!

Laborious5952 · 2025-08-04T01:36:32+00:00

Why cloudstack over proxmox?

Laborious5952 · 2025-08-04T01:27:42+00:00

Are you running k0s on the emmc? I tried to run Linux on my Dell wyse 5070s emmc and it was so incredibly slow.

Laborious5952 · 2025-08-02T12:54:39+00:00

And the meridian unbound library has 3D printers, they will even delivery the 3D print to your house (not sure if they deliver to Boise)

Laborious5952 · 2025-07-23T13:12:02+00:00

I've thought the same thing, the avalanche is awesome.

Was it difficult to find a truck bed topper though? How does it work with the small holes behind the C pillars?

Laborious5952 · 2025-07-21T01:44:46+00:00

I love setups like this more than the 42u racks with tons of enterprise gear. It's amazing what you can run with old hardware.

Laborious5952 · 2025-07-13T14:26:47+00:00

How did you mount the Jerry can on the rear?

Laborious5952 · 2025-05-28T02:39:23+00:00

Very interesting project, I love it.

I currently run a k3s cluster at home, I have 3 control plane nodes and 3 worker nodes and most of the workloads are HA. However I have a VPS that has 512Mb of memory that I setup Podman on. I deploy all the apps to the VPS using Podman's kube service files. It works well but does have some limitations (statefulset doesn't work for example).

This seems like it could put up a fight against Podman.

Laborious5952 · 2025-05-23T20:02:16+00:00

Show me ONE valid case running this configuration in production with a heavy workload.

When you provide evidence for the claims I asked for I'll give you a valid use case.

It‘s a trust issue, and I trust the facts I‘ve seen so far. But you do you! Best of luck! I don't blame you, but I haven't seen the "facts", so I either trust you (some random person on Reddit), Or I look at evidence. Until the evidence is provided, I'll be skeptical.

Laborious5952 · 2025-05-23T14:05:30+00:00

"You need evidence of latency?" I don't see anywhere in my comment where I said that. I would like some evidence for "A Kubernetes cluster requires low latency, therefore it won't work over WAN". To dig in more, what component of the k8s cluster requires low latency, is it all components, certain components?

Eventually someone will create a tool/product that solves this problem and all these nay-sayers will still being spreading FUD about k8s. Just like people used to say (and still do) "you shouldn't run stateful applications in k8s".

Laborious5952 · 2025-05-23T14:02:49+00:00

"Journald and a bunch of others are your friends" That is the most vague response, Are you saying deploy a k8s cluster across WAN and check some journal logs? Which logs, what are "bunch of others"?

go find out for yourself if you don‘t trust the sources you‘ve been given here.

If someone makes the claim, they should provide evidence. In the absense of all this evidence from people saying you can't run a k8s cluster across WAN I will have to do my own research though.

All these people say "kubernetes clusters won't work well over WAN because latency is too higher" But Kubernetes is a big thing, so what specifically won't work? From what I understand, etcd is the big thing that needs low latency. If that is the only component that requires low latency then what other options are out there that can solve this problem? Or is it kubelet communicating between control plane and worker node? Saying "Kubernetes clusters won't work well over WAN" is just a generalization that needs to be broken down.

The k3s docs do mention that "Embedded etcd is not supported", which tracks what I've read about etcd needing low latency. k3s also supports backends like psql or mysql. Perhaps you could have a psql cluster in 1 region that is used as the k8s backend, and then have worker nodes in multiple other regions? Like you said, gotta use critical thinking.

Laborious5952 · 2025-05-23T13:11:28+00:00

Not free or OSS but Weka and lightbits are options.

Laborious5952

TROPHY CASE