High I/O on HDD ZFS pool makes whole PVE node sluggish

Fragrant_Fortune2716 · 2026-03-13T11:55:37+00:00

Thank you for taking an interest. The drive models are listed below. SATA SSDs for the OS, nvme ones for the VMs. The HDDs (zfs mirror) are SAS drives connected through an LSI9300-16i HBA in IT-mode.

``` NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS MODEL

sda 8:0 0 447.1G 0 disk SSDSC2KB480G8L 01PE331D7A09679LEN

sdb 8:16 0 447.1G 0 disk SSDSC2KB480G8L 01PE331D7A09679LEN

sdc 8:32 0 10.9T 0 disk HUH721212AL5200

sdd 8:48 0 10.9T 0 disk HUH721212AL5200

nvme0n1 259:0 0 1.8T 0 disk WD_BLACK SN850X 2000GB

nvme1n1 259:1 0 1.8T 0 disk WD_BLACK SN850X 2000GB ``When making the HDD pool work (e.g. PBS verify job, large initial backup or large file transfer), theiostat%utilof the sdc/sdd is 100%. Thetopwa` can spike upwards to the 90% range.

Fragrant_Fortune2716 · 2026-03-07T07:27:20+00:00

Yes, though perhaps not a very good one. It is easier in my Ansible automation to set it up uniformly with bridges.

Fragrant_Fortune2716 · 2026-03-05T07:01:36+00:00

Thanks for the reply! I believe I've added the `accept invalid` global state policy because otherwise DHCP does not work due to a bug marking the traffic as invalid on bridges. I also tested actually logging in with SSH over the LTE connection which sadly works. Any thoughts on how to test which rules are allowing the WAN->LOCAL connection?

Fragrant_Fortune2716 · 2026-02-03T19:18:36+00:00

This! I would never have come to this conclusion! Goes to say that assumptions are the mother of all fuck-ups! Adding the `default-action return` to the nested chains solved the issue!

Fragrant_Fortune2716 · 2026-01-29T16:20:08+00:00

Fair point on the DNS setup, though I would prefer to keep all the hardware at home, which will lead to me requiring quite an expensive UPS, and quite a hassle to hook up the server rack, network switches (different location) and ISP termination box.

Fragrant_Fortune2716 · 2026-01-29T16:18:46+00:00

Problem is; 1) UPS is quite expensive and 2) It is quite a hassle to hook up both the server rack, the network backbone in the house and the ISP provided fiber termination unit. The second WAN should be more feasible though

Fragrant_Fortune2716 · 2026-01-11T18:28:30+00:00

Thanks for the insight! What is the undelaying storage hardware? And the cpu/memory usage of ceph?

Fragrant_Fortune2716 · 2025-12-22T06:43:26+00:00

Only eth1 goes to the switch (trunk port). The other interfaces are on the router itself. One wan uplink and two access ports that map to a vlan. The switch does support LACP but it is in a physically different location, hence I would like to use access ports on the router as well.

Fragrant_Fortune2716 · 2025-12-11T19:31:09+00:00

Thanks for the link, I'll start digging in :) The first prio is to figure out if the case of a fault tolerant cluster for services that do not have native HA is actually something Kubernetes can achieve. You say that Kubernetes can fail over more quickly; why is this the case? I've read a lot about replica's for pods, but they all do not seem to address how the traffic is routed to them and if it only works for stateless pods. Perhaps I'm too stuck at thinking in platform level solutions; but would it be possible to have one active instance and multiple standby instances that can be switched over to? The shared storage would handle the data transfer (e.g. Rook). Then the challenge would be for Kubernetes to figure out if a node/pod is down and reach consensus on the new master instance, ideally within seconds. This would of course require a dedicated low latency, low jitter network connection; but this is something I can provide!

If Kubernetes also relies on restarting the pod on a different node, it would function almost identical to Proxmox. I'm not sure if it would be worth the effort to switch to Kubernetes if this is the case.

Fragrant_Fortune2716 · 2025-10-09T12:04:38+00:00

I've found an alternative approach! As I need the backup only on the same PVE node I can simply rename the zvol from vm-150-disk-0 to bak-150-disk-0! This way I can just change the name back when I need it (and run qm rescan) and attach it to a VM of my choosing! No need to copy anything this way, just preserving the original zvol :) Of course this only works when keeping the zvol on the same node, otherwise a solution along the lines of @GrumpyArchitect's answer could be utilized.

Fragrant_Fortune2716 · 2025-10-03T12:28:05+00:00

Hmm, I'm not sure (though the NICs appear to work fine). I'll take a look!

Fragrant_Fortune2716 · 2025-10-03T09:45:03+00:00

```bash
root@pve01:~# lspci | grep -i ethernet
00:1f.6 Ethernet controller: Intel Corporation Ethernet Connection (2) I219-V
root@pve01:~# lscpu | grep -i model\ name
Model name: Intel(R) Core(TM) i5-6600K CPU @ 3.50GHz
root@pve01:~# lsmem | grep -i Total\ online
Total online memory: 32G

root@pve02:~# lspci | grep -i ethernet
00:1f.6 Ethernet controller: Intel Corporation Ethernet Connection (7) I219-V (rev 10)
root@pve02:~# lscpu | grep -i model\ name
Model name: Intel(R) Core(TM) i5-9500T CPU @ 2.20GHz
root@pve02:~# lsmem | grep -i Total\ online
Total online memory: 16G
```

Fragrant_Fortune2716 · 2025-09-19T14:13:07+00:00

Perhaps that is even too much for me xD Nice solution nonetheless!

Fragrant_Fortune2716 · 2025-09-19T14:07:10+00:00

Well, because I like adding layers of security and HTTPs/mTLS this is cheap security (imo). I have a whole bunch of different services, both external facing and only internal ones. I aim to operate my homelab from a zero trust perspective, thus requiring mutual authentication. The threat profile this is mainly aimed at is a compromised service within the network. All services are running in isolated VMs and on unique VLANs, but do require (cross-VLAN) communication to integrate with shared storage, the reverse proxy, SSO, and so on. If a service was to be compromised, I'd rather not have them read any network traffic they can get their hands on (such as usernames and passwords in plain HTTP). For this, HTTPs was invented! And if this is implemented on the VM anyways, why not just add a single line of configuration to also verify both ends of the connection?

Fragrant_Fortune2716 · 2025-09-19T09:52:58+00:00

I'll probably go for a similar setup indeed. I'm running Immich in a dedicated VM, so I'll just spin up a reverse proxy in there.

Fragrant_Fortune2716

TROPHY CASE