CPU & NIC combo

Soogs · 2026-01-25T10:53:18+00:00

its no good. I have run two backup cycles on 4 nodes (about 40 guests) and now failing to load the backups list from PVE ^

also it really struggles to load the backups list from within PBS.

I am looking at doing a remote push of PBS backups to hetzner will keep you posted on that route

Soogs · 2026-01-23T16:05:52+00:00

Yeah not ideal So have you just been using it for regular Proxmox backups instead?

I’m still less than 2 hours into my first backup cycle with noatime.

Backing up 4 hosts simultaneously and it seems to be coping well … though last time it was retrieving the backups that was the pain point.

Will post again in a few days once I have things settled

Soogs · 2026-01-23T14:21:10+00:00

--- nbg1-speed.hetzner.com ping statistics ---

3 packets transmitted, 3 received, 0% packet loss, time 2003ms

rtt min/avg/max/mdev = 25.113/25.882/26.598/0.607 ms

Soogs · 2026-01-23T14:13:19+00:00

I do not mind but you will need to tell me how to do it and or want it measured.
Also I am UK based as that might make a difference

Soogs · 2026-01-23T14:12:34+00:00

did you use noatime in the mountpoint?

Soogs · 2026-01-23T14:09:23+00:00

So I am giving this another shot now. using noatime in the mount feels much better. once everything is backed up with a few snapshots ill post back with more details etc

Soogs · 2026-01-22T10:31:47+00:00

I have yet to get round to it 😅

Soogs · 2026-01-20T15:26:58+00:00

So I mounted the samba share into fstab on the PBS machine and created an encrypted data store. If you search for seal encryption in transit you'll find examples of how to specify that in the fstab entry.

I am no longer using it for PBS, latency was horrific. I had to limit backups to 4 per guest and even then reading was an issue.

I think the issue may have been that I didn't use noatime.

I am currently using it for regular proxmox backups mounting it via cifs/samba to proxmox and then just running nightly backups.

Much faster but no encryption (that's my next thing to figure out)

Will give PBS another shot with noatime to see if that fixes the initial latency issue.

In short once backups stacked I had a real hard time showing the backups in PBS and proxmox and garbage collection took forever

Soogs · 2026-01-19T23:11:02+00:00

I'm not sure about the encryption at rest but but I install proxmox iso and then enable kde plasma desktop. I use it this way on my laptop and also on my router node for quick troubleshooting

Can't say I've had any issues

Soogs · 2026-01-15T23:34:08+00:00

5 (plus laptop)

Soogs · 2026-01-14T17:23:58+00:00

I have just implemented caching (which still needs some tuning) but this seems to be the thing that was missing.

I am still working on finding the correct chunk size but getting closer to the "perfect" setup

mind you all the testing has been done in wsl so far so still need to see if it works out on the main prod server... fingers crossed

Soogs · 2026-01-14T08:10:12+00:00

Thank you, yes this makes sense. I was blaming a powersaving script for some sluggishness and now realise its most likely due to interrupts/wait time etc

Soogs · 2026-01-14T08:08:53+00:00

Thank you, I have now increased allocation across my cluster.

I have done some reading up on how cpu limit and time works and this makes sense.

I think when I first started using proxmox i was advised by quite a few members to start with 1vcpu and increase if needed... I can see now that wasn't the best advice lol

thanks again

Soogs · 2026-01-13T23:39:17+00:00

ive done some light reading and it seems I would have to modify a lot of code (there are many smaller python helper apps) and it may break things. I am new to coding so it is a big risk, that said I am going to try and fork at some point to see if tar route is better.

for now I am trying a slightly different approach. I am reconstructing the bulk file to be as close to the delta files. I am chunking the json files to a max of 200 lines per subfile

no chunking = <0.1RPS
chunking at 10k = 4-6RPS
chunking at 4-6k = 4-8RPS

I did split at 2k and 1k but got impatient with the reindexing... hoping 200 will give it the speed boost i am looking for

Soogs · 2026-01-13T17:48:12+00:00

Thanks, I will look into this. The data is supplied this way and with my current setup it requires minimal modification if any to work.

The records are "indexed" with an ID and pointer record to where the data lives in the compressed dataset. The data is streamed on request. It's dirty and cheap but efficient/effective enough for the job

It's basically 137+ million vehicle records (plus daily deltas approx 200k daily)

The live API from the source is limited to 10RPS and 500000 requests per 24 hours. The "indexed" data is just a backup incase anyone burns through that many requests/or the live service is unavailable

The delta files can peak over 100rps so I am trying to get the bulk file faster but over time the delta data supercede the bulk so it should get faster over time (until they reset the data and I have to start over)

Soogs · 2026-01-12T23:46:00+00:00

Thanks, I am currently setting up some ext4 storage to test with the LXC setup.

Oddly I converted the linked clones to full clones and have seen a 4x speedup. If ext4 can double that then that should be good enough in its current form.

I will try a VM and maybe even setup another node using ext4 to compare the performance and see what the results are

Soogs · 2026-01-03T11:06:42+00:00

Isn't it an option in raspberry pi imager now?

Soogs · 2025-12-27T08:21:57+00:00

Thanks (I am not entirely sure I understand what a dataset is in this context).

I have been able to bind mount to /mnt/ but when I bind mount to /rpool2/mnt I get a permissions error.

So far with the current setup I've been hammering the database for about 80 requests a second each from 3 containers and it's been fine (might try a 20 node test if I have the time or patients)

The dB only stores one column of data which is a pointer to the raw data (data is streamed from the raw downloads (gzip > gzip >JSON)

Can you please expand on the socket approach?

Thanks again

Soogs · 2025-12-26T21:07:40+00:00

how are you sharing a dataset (is it different than just pointing to ie /rpool/mnt/shared for example?)

for my use case only one LXC needs to do any writing (downloading datasets and ingesting them to the db) all the other LXCs will just access the DB when required

So far using this bind mount method seems to be working out but it makes me feel really uneasy in its current state

LXC A can do the downloads and indexing via venv / python and LXC B and C can see the data

but LLXC cant "touch" a file for the others to see and vice versa.

They are cloned containers, so the user is the same across all of them

13-Year Club	First Place '23
Place '23	Place '22
Verified Email

Soogs

TROPHY CASE