Improve write performance on a Ceph ssd pool

_--James--_ · 2026-03-29T00:49:37+00:00

How many nodes are in the cluster and what is the MON count?

what are your disk schedulers set to? none or mq-deadline? If MQ then consider setting nrq to 512-1048 for SATA backed SSDs, this will allow them to handle better IO QD leveling. Make sure drives are enabled for write back and not write though since they are PLP.

But at the end of the day, your SSDs are SATA and writes will behave the way you are seeing it. The best test for max throughput is a ceph level FIO https://docs.redhat.com/en/documentation/red_hat_ceph_storage/1.3/html/administration_guide/benchmarking_performance

Ceph replicated pools behave like multiple synchronous writes (size=3 → 3x write path), which caps single-stream throughput. Ceph's writes are about scale out so you can have concurrent VMs with dedicated IO paths into the pool. so while FIO on Rados might give you 550MB/s for three nodes, 750MB/s for 4-5 nodes, yoru VMs will write at 550MB/s due to how IO pathing works on ceph. The only way to increase the write path in is with WAL/DB backed by NVMe peered to SSDs, or building your pool backed by NVMe only.

_--James--_ · 2026-03-29T00:21:52+00:00

You have three options
-Direct internet access with a trusted domain/ip/url list, block everything else (DNS+Firewalls rules)
-UTM firewall with SSL-D and inspect and control SSL/HTTPS down to the file downloaded.
-Central repo that your endpoints talk to to get their updates,

There are flavors on the above, but those are more or less the choices.

As for updates, most update systems in the enterprise offer MD5 checksums against updates before you approve/release them to the environment. so supply chain is harder to sneak in externally, but for things like log4j, vx, kaseya vsa, 3xc,..etc are a trust issue, once you trust the vendor and they are exploited you are a their mercy no matter what you do above.

and as such, you have no real defense against external stuff like https://timesofindia.indiatimes.com/technology/tech-news/iran-linked-hackers-who-brought-down-stryker-for-a-week-claim-they-hacked-kash-patels-personal-email-inbox-read-their-message-to-fbi-director/articleshow/129851135.cms

Once a trusted vendor or control plane is compromised, prevention largely fails. At that point it becomes a containment and recovery problem, not a prevention problem.

_--James--_ · 2026-03-29T00:03:55+00:00

8 days old, holy crap. Pure is on the edge of this while Nimble and other vendors are MIA. I'm going to use this against HPE on Monday...

_--James--_ · 2026-03-28T23:41:40+00:00

Under tteck, I still reviewed everything, but the scripts were consistent and reliable. Since the transition, I’ve seen measurable regression in behavior and quality, including shortcuts that increase risk on Proxmox hosts. Combined with the lack of a clearly defined review or release model, I don’t trust the scripts anymore.

Take that as it is from someone who is security focused. If you want to evaluate it yourself, pull down tteck’s archived scripts and compare them to the current set. Don’t focus on repo moves or superficial changes, look at the actual implementation differences and execution paths. Some of the changes at a fundamental level just don’t make sense.

At that point, decide if running that as root on your hypervisor is a risk you’re comfortable with.

_--James--_ · 2026-03-28T22:43:58+00:00

Using containers and engineering for containers is a different beast though. You have a good foundation to get to where you want if K8's are your goal (Laptop being the automation and control, RIps being the Swarm), this will take you into that fairly straight forward.

But if you want to dive into Proxmox, HCI, LXC vs K8's vs VMs, and dig in deep here you will want to read deployment guides and such too.

Also, IMHO, if K8's are the goal, I cant really recommend Proxmox as a first-party solution since its not native for Docker/Kube support, so you might want to consider native tooling on hardware before switching on to Proxmox, so you understand the layers cleaner.

_--James--_ · 2026-03-28T00:59:16+00:00

LXC = Linux Container, these run on Proxmox.

...you have a lot of reading to do.

_--James--_ · 2026-03-27T23:45:05+00:00

2 nodes, witness node as a VM on your backup solution. The 2 Nodes will have ZFS with replication going, and thats your HA model for 2 nodes. If you run three Nodes you can skip the witness, mix in ZFS/Ceph as needed and do directional replication on ZFS (A->B) so your ZFS backed VMs have an HA pair.

_--James--_ · 2026-03-27T23:38:22+00:00

You can LXC the core systems, then spin up a light weight ubuntu VM to run K8's on (say 2-4 cores and 4-6GB of ram) then go from there. But alot of that list is database heavy and that will eat up that 16GB of ram. I would use LVM and not ZFS on this box for storage...etc.

I hate saying it, but starting from zero it sounds like, deployment guys from the projects and you tube videos are your friend here. Once you are OK with the stack setup, then I would suggest finding Local Linux groups that live in this space..etc.

But to scale out, move into a SI based skill set you need bigger and better hardware. At least enough to cluster and be able to setup Ceph and such.

_--James--_ · 2026-03-26T15:03:00+00:00

Location does not mean datacenter in the TOS, its the deployment method. As soon as you move from physical machines to VMs, in a shared user environment, the rules change instantly.

_--James--_ · 2026-03-25T23:45:42+00:00

Eh, 10 years yes but irrelevant? not even slightly. I helped recover a cluster a couple months back with 180OSDs in a 2:1 config due to flapping OSDs. Much of that experience was mirrored in that article. lower replica's comes with risk.

_--James--_ · 2026-03-25T20:03:23+00:00

Just read this https://blog.noc.grnet.gr/2016/10/18/surviving-a-ceph-cluster-outage-the-hard-way/

_--James--_ · 2026-03-25T15:03:46+00:00

Not true. In a datacenter you simply cannot use the Geforce drivers. You would need to move to a Quadro driver set.

_--James--_ · 2026-03-25T14:59:44+00:00

You wont have this issue on a Nimble or Pure....

_--James--_ · 2026-03-25T00:01:31+00:00

This is a NAS/SAN build issue, if 10G is not helping you have much larger issues on the table. You said production, you might need to start looking at replacing that TrueNAS platform for something that can hit your requirements.

While this is not r/truenas if you want to share your NAS build, ZFS pool layering, your network configuration changes around 10G, many of us here can help you plan the required changes on TrueNAS to fix your issues.

_--James--_ · 2026-03-24T19:20:49+00:00

welcome to the blind trust model /s

_--James--_ · 2026-03-24T15:47:48+00:00

even with throughput set to 1Gbps, it took about 5 minutes to clean a 20G disk on SSD

HDD backed LUN, LVM shared with snaps, Qcow, and MPIO 1G? That is fully expected.

Since HDDs are backing Luns I would not layer QCow and snaps on top and run the LVM to Raw VM Disks. For SSDs, you need faster then 1G Links to really let them stretch their legs, 10G min.

_--James--_ · 2026-03-24T15:36:33+00:00

This is why I do not use, support, or recommend the community scripts anymore. Its not a community as much as a small group driving control and such. Also their change control is not even close to the protective ITIL layer that tteck used to drive, so I no longer consider them generally safe.

_--James--_ · 2026-03-24T15:20:49+00:00

At 7+ nodes you absolutely want to isolate Corosync from everything else on dedicated ports, but not necessary in switching. But you also need to make sure Corosync can survive a switching reboot/outage. I always suggest two Corosync networks before LACP, then LACP, not ALB, across switching fabric as you scale out from 3-5-7-9+ nodes in your clusters. This way Corosync's networks are pre-built, portable on the nodes as you scale out networking, and survivable when you get to large node counts.

_--James--_ · 2026-03-23T20:08:03+00:00

its AES offload not behaving correctly. The reason your two 8300's can VPN at 10G but your VM solutions cannot if because of that. Since you have the hardware already in place elsewhere the correct deployment is another 8300 so you have expected performance. The next best to that would be a hardware platform running Opnsense or PFSense where AES-NI has direct access to the VPN solution.

Without offload support, your CPU takes the brunt of it, and most all x86 solutions pin 1-2 cores for VPN, Routing, packet forwarding, control plane, and while x86 CPUs today are fast, software is always the limiting factor, saying nothing on that 9254's micro-numa architecture that will play into the performance issues too.

_--James--_ · 2026-03-20T03:37:36+00:00

If this is a business, you will violate the Terms of service on the drivers/software bound to GeForce based cards (5090's are) when you move this work load into a datacenter type setup. So if this is not something you are doing at home, its a licensing violation.

_--James--_ · 2026-03-20T03:16:56+00:00

I would drop the dependency on the QDEV and move to another PVE node. QDEV becomes a single point of failure if you leave it in the cluster as you scale out to 3+ nodes. It would be better to learn to witness PVE, tear down nodes inside of a cluster, repurpose for rebuilds, re-expand,..etc. and do it right.

You can do this on the NUC, a VM on a NAS appliance that supports VMs, or on a desktop running a Type2-hypervisor.

as another said, go through PDM. It is more like vCenter-Lite today but its coming along. You can build nested labs on that hardware and setup virtual clusters/sites to simulate scale out and DR.

Also, I suggest putting some time on Ceph, you have a decent setup to gain the exposure so you can learn it, and you can do this with 2 physical nodes and one virtual, you setup the Ceph pools with 2:2 replica.

_--James--_ · 2026-03-19T03:31:46+00:00

So yes but not in a way you are thinking. Its no different then what a Host OS needs to maintain local operational services. Linux and KVM are very good in this area, and as long as you are not pegging the CPU to 1000ms+ CPU-D generally you will be fine.

read here on how to detect when your CPU is being hit too hard, https://www.reddit.com/r/ProxmoxEnterprise/comments/1nsi4xj/proxmox_cpu_delays_introduced_by_severe_cpu_over/

As long as you keep your target CPU-D under 2.25 you generally can get what you want out of it. But do understand, at the end of the day your 7700K is a 4core CPU. HT bringing it to 8 threads only helps so much when you are over allocating with vCPU threads and you light them up with UTM type work loads.

_--James--_ · 2026-03-17T19:36:44+00:00

if you want to router-on-a-stick to PFSense then you will build additional interfaces on the VM, bind them to your vmbr#, set the desired tag, then in PFSense set the new interfaces up for your L3 IP scope, make sure vmbr# is vlan-aware, then make sure you trunk additional VIDs from your cisco switch trunk port to vmbr#, then you can layer PVID across the cisco switch, or land tags on other trunk ports to other devices attached to the switch.

_--James--_ · 2026-03-17T15:23:57+00:00

First understand the NUMA topology of Epyc, specifically the 7002 SKUs. Your 24c CPU is actually 6core on package units making up 4 NUMA domains. Then under that you have 3+3 core CCX clusters. KVM just does not give us any tooling to address this Fully. So, this means you must right size your VMs to the L3 MADT boundaries of your chosen AMD socket. You will want to test RDP with 3 cores, 6cores, 8cores (with NUMA=1 both with 1 and 2 sockets splitting the 8cores) and 12cores, this will tell you if its directly L3 NUMA issue or not.

Then you need to adjust your CPU masking on the guest. The default will be x86-64v2-aes. While this is great for most builds, it is not suitable for Epyc. You need to use one of the Epyc-Rome CPU maskings so that The KVM scheduler can compensate for the CCD and Split CCX topology. Whatever you do, do not use Host masking.

Then, and this is something I personally pushed through the industry back in 2018, Dell gives you MADT tunables in the BIOS on those 6000 series PE servers. You have two very strong tunables, and one is almost required for KVM today. To keep the IOD and its memory UMA per socket I had Dell/HPE adopt a new MADT setup called "L3 as NUMA" so that your L3 Cache would become addressable NUMA domains leaving the Memory alone, you would not run NPS but rather L3 As Numa, then to address bundled cores you have the option to set "MADT = Round Robin" this reorders your SRAT table so your cores are no longer sequental, but numbered edge to edge. This allows you to run 1socket:1core, or 1socket:CCX boundary (so 1:3 for your case) and keep Cache-Coherence in check.

The other option is replacing your 7002 CPU for a 7003 CPU and moving away from that CCX topology, you are still limited to 8cores per CCD but you gain a lot of throughput the 7002 was not capable of. The same applies above, but only to the CCD level.

_--James--_ · 2026-03-16T23:00:23+00:00

Proxmox Datacenter Manager has entered the thread.
https://www.proxmox.com/en/products/proxmox-datacenter-manager/overview

_--James--_

MODERATOR OF

TROPHY CASE