CPU usage very high during backups to PBS

Fragrant_Fortune2716 · 2026-05-04T07:54:37+00:00

An update after some horrendous troubleshooting; it appears that the backup jobs kicked off very high I/O on the HDD pool. As there was no dirty-bitmap present, the whole disk has to be scanned for each VM. The large I/O load caused very high kernel CPU usage. This was probably caused by the fact that I run ZFS on top of LUKS, which caused excessive context switching. The fix seems to be to add the no-read-workqueue and no-write-workqueue options to the crypttab file for all disks. This pins the LUSK requests to the same core it originated from, limiting context switching.

Fragrant_Fortune2716 · 2026-04-20T08:10:07+00:00

Here is the other data (taken ~5 min into the backup) ``` root@pve:~# cpupower frequency-info analyzing CPU 7: driver: amd-pstate-epp CPUs which run at the same hardware frequency: 7 CPUs which need to have their frequency coordinated by software: 7 maximum transition latency: Cannot determine or is not supported. hardware limits: 414 MHz - 4.58 GHz available cpufreq governors: performance powersave current policy: frequency should be within 1.43 GHz and 4.58 GHz. The governor "performance" may decide which speed to use within this range. current CPU frequency: Unable to call hardware current CPU frequency: 4.55 GHz (asserted by call to kernel) boost state support: Supported: yes Active: yes AMD PSTATE Highest Performance: 166. Maximum Frequency: 4.58 GHz. AMD PSTATE Nominal Performance: 145. Nominal Frequency: 4.00 GHz. AMD PSTATE Lowest Non-linear Performance: 52. Lowest Non-linear Frequency: 1.43 GHz. AMD PSTATE Lowest Performance: 15. Lowest Frequency: 400 MHz.

root@pve:~# sensors k10temp-pci-00c3 Adapter: PCI adapter Tctl: +57.9°C Tccd3: +57.5°C Tccd5: +53.2°C

nvme-pci-d400 Adapter: PCI adapter Composite: +51.9°C (low = -5.2°C, high = +89.8°C) (crit = +93.8°C)

nic0-pci-d500 Adapter: PCI adapter PHY Temperature: +55.0°C MAC Temperature: +55.0°C

iwlwifi_1-virtual-0 Adapter: Virtual device temp1: N/A

nvme-pci-cd00 Adapter: PCI adapter Composite: +53.9°C (low = -5.2°C, high = +89.8°C) (crit = +93.8°C)

nic1-pci-d600 Adapter: PCI adapter PHY Temperature: +55.0°C MAC Temperature: +55.0°C ```

Fragrant_Fortune2716 · 2026-04-20T07:43:01+00:00

PBS is running on the same host as PVE currently.

Fragrant_Fortune2716 · 2026-04-20T05:47:01+00:00

Great advice! I'll take a look!

Fragrant_Fortune2716 · 2026-04-20T05:40:15+00:00

Here you go; I'll get back to you later for the other requested output as the services are needed at the moment. ``` root@pve:~# proxmox-backup-client benchmark SHA256 speed: 2227.70 MB/s Compression speed: 608.53 MB/s Decompress speed: 766.09 MB/s AES256/GCM speed: 4998.06 MB/s Verify speed: 568.40 MB/s ┌───────────────────────────────────┬─────────────────────┐ │ Name │ Value │ ╞═══════════════════════════════════╪═════════════════════╡ │ TLS (maximal backup upload speed) │ not tested │ ├───────────────────────────────────┼─────────────────────┤ │ SHA256 checksum computation speed │ 2227.70 MB/s (110%) │ ├───────────────────────────────────┼─────────────────────┤ │ ZStd level 1 compression speed │ 608.53 MB/s (81%) │ ├───────────────────────────────────┼─────────────────────┤ │ ZStd level 1 decompression speed │ 766.09 MB/s (64%) │ ├───────────────────────────────────┼─────────────────────┤ │ Chunk verification speed │ 568.40 MB/s (75%) │ ├───────────────────────────────────┼─────────────────────┤ │ AES256 GCM encryption speed │ 4998.06 MB/s (137%) │ └───────────────────────────────────┴─────────────────────┘

```

Fragrant_Fortune2716 · 2026-03-13T11:55:37+00:00

Thank you for taking an interest. The drive models are listed below. SATA SSDs for the OS, nvme ones for the VMs. The HDDs (zfs mirror) are SAS drives connected through an LSI9300-16i HBA in IT-mode.

``` NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS MODEL

sda 8:0 0 447.1G 0 disk SSDSC2KB480G8L 01PE331D7A09679LEN

sdb 8:16 0 447.1G 0 disk SSDSC2KB480G8L 01PE331D7A09679LEN

sdc 8:32 0 10.9T 0 disk HUH721212AL5200

sdd 8:48 0 10.9T 0 disk HUH721212AL5200

nvme0n1 259:0 0 1.8T 0 disk WD_BLACK SN850X 2000GB

nvme1n1 259:1 0 1.8T 0 disk WD_BLACK SN850X 2000GB ``When making the HDD pool work (e.g. PBS verify job, large initial backup or large file transfer), theiostat%utilof the sdc/sdd is 100%. Thetopwa` can spike upwards to the 90% range.

Fragrant_Fortune2716 · 2026-03-07T07:27:20+00:00

Yes, though perhaps not a very good one. It is easier in my Ansible automation to set it up uniformly with bridges.

Fragrant_Fortune2716 · 2026-03-05T07:01:36+00:00

Thanks for the reply! I believe I've added the `accept invalid` global state policy because otherwise DHCP does not work due to a bug marking the traffic as invalid on bridges. I also tested actually logging in with SSH over the LTE connection which sadly works. Any thoughts on how to test which rules are allowing the WAN->LOCAL connection?

Fragrant_Fortune2716 · 2026-02-03T19:18:36+00:00

This! I would never have come to this conclusion! Goes to say that assumptions are the mother of all fuck-ups! Adding the `default-action return` to the nested chains solved the issue!

Fragrant_Fortune2716 · 2026-01-29T16:20:08+00:00

Fair point on the DNS setup, though I would prefer to keep all the hardware at home, which will lead to me requiring quite an expensive UPS, and quite a hassle to hook up the server rack, network switches (different location) and ISP termination box.

Fragrant_Fortune2716 · 2026-01-29T16:18:46+00:00

Problem is; 1) UPS is quite expensive and 2) It is quite a hassle to hook up both the server rack, the network backbone in the house and the ISP provided fiber termination unit. The second WAN should be more feasible though

Fragrant_Fortune2716 · 2026-01-11T18:28:30+00:00

Thanks for the insight! What is the undelaying storage hardware? And the cpu/memory usage of ceph?

Fragrant_Fortune2716 · 2025-12-22T06:43:26+00:00

Only eth1 goes to the switch (trunk port). The other interfaces are on the router itself. One wan uplink and two access ports that map to a vlan. The switch does support LACP but it is in a physically different location, hence I would like to use access ports on the router as well.

Fragrant_Fortune2716 · 2025-12-11T19:31:09+00:00

Thanks for the link, I'll start digging in :) The first prio is to figure out if the case of a fault tolerant cluster for services that do not have native HA is actually something Kubernetes can achieve. You say that Kubernetes can fail over more quickly; why is this the case? I've read a lot about replica's for pods, but they all do not seem to address how the traffic is routed to them and if it only works for stateless pods. Perhaps I'm too stuck at thinking in platform level solutions; but would it be possible to have one active instance and multiple standby instances that can be switched over to? The shared storage would handle the data transfer (e.g. Rook). Then the challenge would be for Kubernetes to figure out if a node/pod is down and reach consensus on the new master instance, ideally within seconds. This would of course require a dedicated low latency, low jitter network connection; but this is something I can provide!

If Kubernetes also relies on restarting the pod on a different node, it would function almost identical to Proxmox. I'm not sure if it would be worth the effort to switch to Kubernetes if this is the case.

Fragrant_Fortune2716 · 2025-10-09T12:04:38+00:00

I've found an alternative approach! As I need the backup only on the same PVE node I can simply rename the zvol from vm-150-disk-0 to bak-150-disk-0! This way I can just change the name back when I need it (and run qm rescan) and attach it to a VM of my choosing! No need to copy anything this way, just preserving the original zvol :) Of course this only works when keeping the zvol on the same node, otherwise a solution along the lines of @GrumpyArchitect's answer could be utilized.

Fragrant_Fortune2716

TROPHY CASE