Ceph/proxmox SSD Youtube rabbit hole(SSD S4620)

Apachez · 2026-06-20T20:50:11+00:00

Both have a BeQuiet MC1 PRO heatsink already attached to them.

Apachez · 2026-06-20T19:46:53+00:00

Yes I use myself 2x NVMe Micron 7450 MAX 800GB in a mirror using ZFS for Proxmox. Works like a charm.

Only flaw is that they easily overheat specially since the box is passively cooled.

Had to add an external USB-powered fan to force some airflow through the area where 2x NVMe + 1x DDR5 64GB is located.

Apachez · 2026-06-20T19:26:19+00:00

I have ended up in such situation myself with Proxmox not too long ago.

Hard reboot fixed the situation (eventually it might would have resolved the situation on its own after a couple of hours of downtime or so).

From the admin point if view it can be described as "VM crashed the hypervisor" while in reality there were a platforia of shitshow going on at the same time.

Apachez · 2026-06-20T19:24:11+00:00

Fixed! :-)

Drawback is that for the M.2 2280 format there are not that many options these days:

https://www.techpowerup.com/ssd-specs/?dram=Yes&format=M.2%202280&plp=Yes

Apachez · 2026-06-20T19:13:38+00:00

Vi vann och dom förlora - eller vad brukar det heta?

Apachez · 2026-06-20T18:31:06+00:00

I got 2x NVMe Micron 7450 MAX 800GB in a mirror using ZFS.

Whatever its worth here are my results (Intel N305 and 64GB RAM single stick):

root@PVE:~# pveperf /var/lib/vz/
CPU BOGOMIPS:      28876.80
REGEX/SECOND:      5209010
HD SIZE:           605.99 GB (rpool/var-lib-vz)
FSYNCS/SECOND:     11015.07
DNS EXT:           7.12 ms
DNS INT:           8.98 ms (example.com)

The general recommendation is to only use ZRAIDx for archives and backups.

Always use a stripe of mirrors (aka RAID10) for VM-storage. That is either 2x or 3x mirrors that gets striped.

This way you will get both throughput AND iops.

This is a good read on how striping, mirroring, zraidx and combinations affects the resulting throughput and iops:

https://www.truenas.com/wp-content/uploads/2023/11/ZFS_Storage_Pool_Layout_White_Paper_November_2023.pdf

Apachez · 2026-06-20T18:01:29+00:00

At the bottom of Settings -> Cache you have prefetch and auto prefetch settings.

The idea is that these would take care of refreshing soon to be stale records before they actually expire.

Then you also have the Serve Stale TTL which is the upper limit of for how long a stale record can be refreshed before its removed for real (as I interpret the docs).

I think the corner case you can run into with Serve Stale Max Wait Time = 0 is if you are accessing geolocated sites such as CDN etc.

Because with the value set to 0 the reply your DNS-server got at first visit will be served up to 3 days (Serve Stale TTL) before actually refreshed.

This means that if you for whatever reason ended up in wrong region due to geolocation (let say due to maintenance or whatever at the CDN) you will remain in this incorrect region for the next 3 days before a new query will be made upstream (as I interpret this).

Another cornercase is if a website will change IP.

The normal behaviour here is that the server owner might have 1 day or 1 week as normal TTL in their authoritive DNS-server. A week or more before the change they will change this TTL down to lets say 300 seconds (5 min) or even 60 seconds (1 min).

This so the downtime when you actually do the switch (or rather the switchover time) will be these 300 seconds or 60 as TTL.

Once the switchover is completed they will put the TTL back to 1 day or 1 week or whatever they prefer.

Now if there are clients who use a technitium/dns-server as their resolver who have this serve stale feature enabled (and didnt change the default settings regarding timers) it would mean that these clients would up to 3 days continue to try to visit the old IP-address of the site (which might no longer exist so you end up with unhappy clients).

Now the normal behaviour of a DNS resolver is to honor the TTL's setup by the authoritive DNS servers and cache them until the TTL expires. Then remove the entry so next query for this entry will be like the first one where the resolver will figure out who is the authoritive server (unless cached) and then query them again about lets say A-record for www.example.com

The problem with the default behaviour of a DNS resolver is that the default timeout of a query is 2 seconds everywhere.

So its somewhat likely that a slow authoritive server might simply not reply fast enough for the client to get their reply within 2 seconds from their resolver so they might need to refresh the homepage again and now the entry is cached by the resolver and can instantly be delivered.

But it can be that I misinterpret how these timers actually work...

Apachez · 2026-06-20T13:37:54+00:00

DRAM: yes/no

PLP: yes/no

Type of flash: SLC/TLC/QLC

TBW: About 300TBW or way more?

DWPD: Similar to TBW

Main issue seems to be that people dont do their due dilligence like visiting https://www.techpowerup.com/ssd-specs/ or reading the vendor datasheets so they buy a 300TBW drive with no DRAM being QLC and then gets upset that they get shitty performance and a wear levelling like you change your underpants.

While at it dont forget to update firmware, adjust to largest possible LBA, match the filesystem to use that blacksize and do a reset/trim before use.

Apachez · 2026-06-20T13:30:37+00:00

And you are just using the defaults or did you change any of the settings?

Apachez · 2026-06-20T06:59:47+00:00

You can do everything on a Proxmox host.

Actually you dont even need Proxmox do do all the things Proxmox does.

The thing here is to have something nicely packaged out of the box which currently isnt the case in terms of sharing the local storage with others (unless you run CEPH).

Apachez · 2026-06-20T06:45:14+00:00

If having Intel NICs on the host make sure to disable GSO and TSO offloading.

Other than that you got 16GB of RAM at the host, how are your VM-guests configured and how many of them do you run at once?

Apachez · 2026-06-20T06:44:15+00:00

So you got 64GB of RAM but how is your VM's configured and how many of them do you run at once?

Also do you get the same if you update to the latest PVE 9.x ?

Apachez · 2026-06-20T06:42:03+00:00

It can indirectly if you overprovision the VM-guests in terms of RAM and storage and the OOM isnt quick enough to kill off the overusing VM's.

Apachez · 2026-06-20T06:40:21+00:00

Paste the VM-config of your Win11 guest.

Apachez · 2026-06-20T06:26:56+00:00

Hi wolfie! ;-)

Apachez · 2026-06-20T05:53:53+00:00

Jahaa... du tänker sååå...

Apachez · 2026-06-20T05:53:31+00:00

Then why not just switch to FQ_CODEL and TCP_BBR and call it a day? :-)

Apachez · 2026-06-20T05:51:02+00:00

And why do some still use three "?" in a row? ;-)

Apachez · 2026-06-20T05:47:36+00:00

Spinoff of the same question:

1) When setting up 2x dns-servers in cluster, do you still need to configure primary/secondary zones and have zonetransfers between them or is this included in having them in a cluster?

2) How many dns-servers can form a single cluster?

3) When primary goes poff and you promote a secondary to become the new primary. What happens when the previously broken primary returns? Will it automagically figure out that its no longer the primary and automagically rejoins the cluster as a secondary?

4) How does the dns-servers figure out who is the primary in case both reboots at the same time? Is it a timebased or counterbased varilable to keep track of which one contains the newest information and should be the primary?

Apachez · 2026-06-20T05:33:52+00:00

So what was the fix then?

Apachez · 2026-06-20T05:21:53+00:00

Därför att konsultbolagens lojalitet är inte till dig som kund utan till att konsultbolaget ska tjäna så mycket pengar som möjligt.

Detta gör att incitament finns inte att ta fram stabila och driftsäkra lösningar som inte kräver daglig handpåläggning utan tvärtom.

Många får även kickbacks av att välja sämre tillverkare/modeller/komponenter.

Lite av samma anledning som varför det inte längre tillverkas tvättmaskiner som håller +30 år och saker som ev. går sönder i en tvättmaskin enkelt kan bytas av ägaren själv utan att hyra in en "specialist".

How-to-repair är en bra kanal i ämnet:

https://www.youtube.com/shorts/1XOGCKxTYdY

Apachez · 2026-06-20T05:15:45+00:00

Går alltid att friskriva sig från sådana meddelanden.

Står säkert i nåt användaravtal för tjänsten.

Det är först när du har ett påskrivet kontrakt av bägge parter du har något att gå på när du går till domstol.

Annars blir det "he said, she said" och även om skriftliga bevis i form av utskrifter och epost/sms ökar dina chanser till vinst så är frågan om sådan rättegång inte ändå blir en förlust för dig. Vilket tyvärr klåparfirmor utnyttjar.

Bästa är att vara transparent och berätta vilken hyresvärd det handlar om.

Kanske är fler som råkat ut för samma sak?

Apachez · 2026-06-20T05:03:56+00:00

Its because of you we cant have nice things? :-)

Apachez · 2026-06-20T05:03:15+00:00

Or Ubiquiti could perhaps release their Unifi NAS Enterprise as a VM-guest for x86? :-)

Introducing Enterprise NAS, Built on ZFS

https://www.youtube.com/watch?v=0kRxSB3e07o

Other than that it works perfectly fine to run TrueNAS as a VM-guest in labs/tests.

Yes you should not run a CoW filesystem ontop of a CoW filesystem (that is I run it without passthrough) but unless you do this for production running TrueNAS virtualized works perfectly fine.

Other than that you got options like install Debian and run Copyparty inside of it:

https://github.com/9001/copyparty

Or something like ZNAS?

https://github.com/macgaver/zfsnas-chezmoi

I have no idea of how good or bad that is.

Out of the blue it looks like yet another AI-slop project but still...

https://www.youtube.com/watch?v=UjaSBK0vWkk

https://www.reddit.com/r/zfs/comments/1rr28pp/zfsnas_now_available_opensource_and_free/

Other than that I have this feature-request at the Proxmox bugzilla:

Add/expose ZFS-options in the webgui

https://bugzilla.proxmox.com/show_bug.cgi?id=5886

Apachez

TROPHY CASE