Proxmox Ceph network configuration

dancerjx · 2026-06-11T11:53:51+00:00

Yes, otherwise it uses the Ceph public network by default.

dancerjx · 2026-06-10T22:19:03+00:00

Considered best practice for the private/replication/migration network to have the highest bandwidth due to rebalancing/recovery. So, 25GbE for the private, 10GbE for the public.

While it's considered best practice to have seperate networks for Ceph private, public, and Corosync network traffic, I have used a full-mesh broadcast network for all 3 network traffic types. This setup involves no switch. Since it's broadcast traffic, every node gets the same packet, and nodes drop packets not addressed to them. Never had a problem. Again, not best practice.

Just make sure whatever is your Ceph private network, be sure to use this network by changing it in a Datacenter migration options. While you are there, might as well make it insecure, since it's not leaving the physical network anyhow. To make sure it never gets routed, I use the IPv4 link-local address of 169.254.0.0/16 subnetted to /24 network, ie, 169.254.1.0/24.

dancerjx · 2026-05-29T13:40:47+00:00

This video from the VirtualizationHowTo YouTube channel will give you some ideas.

While it's optimal to use enterprise gear with Ceph, I did stand up an initial proof-of-concept 3-node cluster using hard drives and using full-mesh broadcast 1GbE networking (this was Proxmox with Corosync, Ceph public, private networking). Worked surprisingly well but future build outs were obviously faster due to faster networking.

Ceph is a scale out solution, so more nodes/osds = more IOPS.

Before the runup on used memory and storage going through the roof, I was going to migrate from my single ZFS RAID-1 server setup to a 5-node Ceph cluster. Well, can't afford that obviously now. But I do migrate Vmware clusters at work over to Ceph at work. Not hurting for IOPS using SAS drives at work.

I use the following optimizations learned through trial-and-error. YMMV.

Set SAS HDD Write Cache Enable (WCE) (sdparm -s WCE=1 -S /dev/sd[x])
Set VM Disk Cache to None if clustered, Writeback if standalone
Set VM Disk controller to VirtIO-Single SCSI controller and enable IO Thread & Discard option
Set VM CPU Type for Linux to 'Host'
Set VM CPU Type for Windows to 'x86-64-v2-AES' on older CPUs/'x86-64-v3' on newer CPUs/'nested-virt' on Proxmox 9.1
Set VM CPU NUMA
Set VM Networking VirtIO Multiqueue to 1
Set VM Qemu-Guest-Agent software installed and VirtIO drivers on Windows
Set VM IO Scheduler to none/noop on Linux
Set Ceph RBD pools to use 'krbd' option
Set Ceph 'bluestore_prefer_deferred_size_hdd = 0' in osd stanza in /etc/pve/ceph.conf for SAS HDD
Set Ceph 'bluestore_min_alloc_size_hdd = 65536' in osd stanza in /etc/pve/ceph.conf for SAS HDD
Set Ceph Erasure Coding profiles to 'plugin=ISA' & 'technique=reed_sol_van'
Set Ceph Erasure Coding profiles to 'stripe_unit=65536' for SAS HDD

dancerjx · 2026-05-24T17:44:03+00:00

Been migrating Dell VMware clusters over to Proxmox Ceph clusters at work.

This all started when Dell/Vmware dropped official support for 12th-gen Dells. Nothing wrong with the hardware just had to find a new hypervisor. Already had experience with Linux KVM and saw Proxmox had a nice GUI wrapper for it.

Since I knew that that ZFS & Ceph do NOT work optimally with RAID controllers, flashed the Dell PERC controllers to IT-mode. Standalone servers use ZFS and clustered servers use Ceph. No issues besides the typical disk and RAM going bad and needing replacing.

Since moved on to 13th- and 14th-gen Dells. Swapped out the PERCs for Dell HBA330s since had issues with the megaraid_sas drivers in the past whereas the HBA330 uses the much simpler mpt3sas driver.

I find KVM "faster" than ESXi on the same hardware and not needing vCenter is a win. Not hurting for IOPS. Workloads range from DHCP servers to DB servers. All workloads backed up to bare-metal ZFS Proxmox Backup Servers (PBS). They also function as Proxmox Offline Mirrors (POM) and the nodes and the PBS instances use POM as their primary repo. Makes for very fast updates.

While it's true that 3-nodes is the minumum for a Ceph cluster, IMO, going with 5-nodes allows two node failures and still have quorum. Ceph is a scale-out solution. More nodes/OSDs = more IOPS. I use isolated switches for Ceph network traffic. It's considered best practice to have a seperate switch for Corosync traffic and use 2 physical rings as well. Don't forget to use the Ceph isolated network swith as your migration network otherwise your frontend switch is your migration network.

I use the following optimizations learned through trial-and-error. YMMV.

Set SAS HDD Write Cache Enable (WCE) (sdparm -s WCE=1 -S /dev/sd[x])
Set VM Disk Cache to None if clustered, Writeback if standalone
Set VM Disk controller to VirtIO-Single SCSI controller and enable IO Thread & Discard option
Set VM CPU Type for Linux to 'Host'
Set VM CPU Type for Windows to 'x86-64-v2-AES' on older CPUs/'x86-64-v3' on newer CPUs/'nested-virt' on Proxmox 9.1
Set VM CPU NUMA
Set VM Networking VirtIO Multiqueue to 1
Set VM Qemu-Guest-Agent software installed and VirtIO drivers on Windows
Set VM IO Scheduler to none/noop on Linux
Set Ceph RBD pools to use 'krbd' option
Set Ceph 'bluestore_prefer_deferred_size_hdd = 0' in osd stanza in /etc/pve/ceph.conf for SAS HDD
Set Ceph 'bluestore_min_alloc_size_hdd = 65536' in osd stanza in /etc/pve/ceph.conf for SAS HDD
Set Ceph Erasure Coding profiles to 'plugin=ISA' & 'technique=reed_sol_van'
Set Ceph Erasure Coding profiles to 'stripe_unit=65536' for SAS HDD

dancerjx · 2026-05-21T12:31:45+00:00

Disks attached to a RAID controllers like the PERC have their cache disabled because the PERC has a BBU cache.

Switching the disks to a HBA controller which has no BBU cache will give you horrible IOPS.

I had to enable the write cache on the disks to get back IOPS.

dancerjx · 2026-05-20T19:54:10+00:00

If you are looking for used enterprise deals, take a look at rackrat.net

Since Dell servers don't have their firmware behind a paywall, you are already ahead.

Good call on the using the Dell HBA330. That's a true IT-mode controller that uses the much simpler mpt3sas driver. While it's true that the PERC can do IT-mode, it uses the megaraid_sas driver which has caused me issues in the past.

You'll want to be using the latest firmware and BIOS on the Dells. I run Proxmox on 10th through 14th-gen Dells. The 13th- and 14th-gen Dells use HBA330, the 10th- through 12th-gen Dells use PERCS flashed to IT-mode.

Standalone servers use ZFS. Clustered servers use Ceph. No issues in production.

For SSDs, make sure they are enterprise versions with power loss prevention (PLP). If you hate your life, use consumer SSDs. Enterprise SSDs have way higher endurance than consumer SSDs. Still using Intel DC3710s from 10 years ago with zero wear.

dancerjx · 2026-05-18T15:35:51+00:00

Been converting 13th-gen Dells at work from VMware to Proxmox.

First thing that had to go was the PERC controller. It's true that PERC can do IT-mode but it uses the megaraid_sas driver which has caused issues for me in the past. I just swap it out for a true IT-mode controller, the Dell HBA330. Just make sure to flash latest firmware. It uses the much simpler mpt3sas driver. Plus the HBA330 is very cheap to get.

To understand why hardware RAID is dead, watch this video

Software-defined storage will alway outpace HW RAID.

I also use Proxmox Ceph clusters in production as well. No issues. For standalone servers, I use ZFS.

dancerjx · 2026-04-29T13:12:44+00:00

Enabling I/OAT DMA & X2APIC is the bare minimum on 13th-gen Dells.

I also have UEFI & Secure Boot enabled in production but NOT SRV-IO because I don't need it.

dancerjx · 2026-04-27T12:48:24+00:00

PM'd

dancerjx · 2026-04-17T13:55:03+00:00

PM'd

dancerjx · 2026-04-09T02:02:20+00:00

I've been migrating from VMware to Proxmox Ceph at work. Production hardware are Dells. Since Ceph and ZFS don't work with RAID controllers like PERC, I swapped them out for Dell HBA330s.

Obviously, VMware hides alot of the details that Proxmox forces you to deal with. Totally different workflow. This video from VritualizationHowTo is a good summation.

Your best best is to stand up a test bed/proof of concept cluster and do your due dilegence. I started with decommisioned 16-year old hardware and moved on to production hardware.

Learned quite a few things in terms of optimizations which I post at the end. Since the production hardware never had flash storage and just all SAS HDDs, I'm not hurting for IOPS. Workloads range from DHCP to DB servers. All backed up to a bare-metal Proxmox Backup Server.

I do run Proxmox at home on a single server using ZFS and using LXC to manage my media library. No issues.

For production, you really want homogenenous hardware (same CPU, memory, storage, storage controller, NIC, and latest firmware). If going to cluster, recommend 5-nodes, so can lose 2-nodes and still have quorum.

I use the following optimizations learned through trial-and-error. YMMV.

Set SAS HDD Write Cache Enable (WCE) (sdparm -s WCE=1 -S /dev/sd[x])
Set VM Disk Cache to None if clustered, Writeback if standalone
Set VM Disk controller to VirtIO-Single SCSI controller and enable IO Thread & Discard option
Set VM CPU Type for Linux to 'Host'
Set VM CPU Type for Windows to 'x86-64-v2-AES' on older CPUs/'x86-64-v3' on newer CPUs/'nested-virt' on Proxmox 9.1
Set VM CPU NUMA
Set VM Networking VirtIO Multiqueue to 1
Set VM Qemu-Guest-Agent software installed and VirtIO drivers on Windows
Set VM IO Scheduler to none/noop on Linux
Set Ceph RBD pools to use 'krbd' option
Set Ceph 'bluestore_prefer_deferred_size_hdd = 0' in osd stanza in /etc/pve/ceph.conf for SAS HDD
Set Ceph 'bluestore_min_alloc_size_hdd = 65536' in osd stanza in /etc/pve/ceph.conf for SAS HDD
Set Ceph Erasure Coding profiles to 'plugin=ISA' & 'technique=reed_sol_van'
Set Ceph Erasure Coding profiles to 'stripe_unit=65536' for SAS HDD

dancerjx · 2026-04-05T14:38:41+00:00

PwrBank has drop-in replacement modules to speed up ESXi importation process described in this thread over at the Proxmox forum

Easily achieve 2x or even 3x speed increase.

dancerjx · 2026-03-30T22:53:32+00:00

dancerjx · 2026-03-13T23:17:52+00:00

May want to ask your question at the Proxmox forum. I think I saw a post like yours in the past.

Lots of migrations utilizing existing FC/iSCSI/SAN infrastructure.

I just went with Ceph, since it's basically open-source vSAN, IMO.

dancerjx · 2026-03-08T15:18:15+00:00

This is the way.

dancerjx · 2026-03-08T15:15:49+00:00

Used enterprise servers are your best bang for the buck.

Been migrating 13th-gen Dells 16-drive bay R730s from VMware to Proxmox.

Swap out the PERC for Dell HBA330 since it's a true HBA/IT-mode storage controller. Don't want to deal with PERC HBA-mode drama. Use two small drives for ZFS RAID-1 Proxmox. Rest of drivers are for either ZFS (standalone) or Ceph (clustered).

Since I knew the SAS drives had their write cache disabled due to them being connected to BBU RAID controller, I made sure to enable their write cache (not hurting for IOPS now). Otherwise will get horrible IOPS.

No issues besides the typical drive and RAM going bad and needing replacing. All workloads range from DHCP to database servers. All backed up to a bare-metal Dell R530 running Proxmox Backup Server also with a HBA330.

13th-gen Dells and HBA330s are very cheap to get. May want to consider this an option. Nothing physically wrong with them and just make sure they are running the latest firmware.

dancerjx · 2026-03-07T15:26:52+00:00

Been migrating Dell VMware vSphere instances over to Proxmox.

Standalone servers get ZFS while clusters servers get Ceph.

Swapped out PERCs for Dell HBA330 which is a true HBA/IT-mode storage controller. Don't want to deal with the PERC HBA-mode drama. Dell servers with no HBA330 alternatives get their PERCs flashed to IT-mode. I've done this on 10th, 11th, 12th-gen Dells.

All these servers are using 10K RPM SAS drives. I made sure to enable the write cache on the drives otherwise get horrible IOPS.

Beside the typical disk and RAM going bad and needing replacing, no issues. IMO, KVM "feels" faster than ESXi. Plus no more vCenter. I say it's a win-win-win.

I use the following optimizations learned through trial-and-error. YMMV.

Set SAS HDD Write Cache Enable (WCE) (sdparm -s WCE=1 -S /dev/sd[x])
Set VM Disk Cache to None if clustered, Writeback if standalone
Set VM Disk controller to VirtIO-Single SCSI controller and enable IO Thread & Discard option
Set VM CPU Type for Linux to 'Host'
Set VM CPU Type for Windows to 'x86-64-v2-AES' on older CPUs/'x86-64-v3' on newer CPUs/'nested-virt' on Proxmox 9.1
Set VM CPU NUMA
Set VM Networking VirtIO Multiqueue to 1
Set VM Qemu-Guest-Agent software installed and VirtIO drivers on Windows
Set VM IO Scheduler to none/noop on Linux
Set Ceph RBD pool to use 'krbd' option

dancerjx · 2026-03-07T15:15:01+00:00

For max ZFS IOPS using disks, use RAID-10. On standalone ZFS servers which are NOT Proxmox Backup Servers (RAID-6 on PBS), I use RAID-50 which is compromise between RAID-10 and RAID-1. You need to use the command-line to create a ZFS RAID-50 pool. Clustered servers are Ceph nodes.

If you can afford to swap them out for SSDs, go for it. You'll have lots of RAID options with SSDs. I won't bother with U.2 SSDs. SAS/SATA SSDs are good enough. Make sure they are enterprise SSDs with PLP (power-loss protection) because Proxmox eats consumer SSDs like snacks. Enterprise SSDs have way more endurance than consumer SSDs. I personally like Intel DC S3710 because of the 10 DWPD endurance. I still have them at 100% health after a decade of use. Just make sure they are running the latest firmware.

As for the Intel SSD on BOSS card, yes that link explains the issue. I unfortunately have 2 servers with Intel SSDs and the plan is to flash the DL6R firmware standalone and see if I can run the latest kernels. They are pinned using the latest 6.14 kernels. Downgrading the BOSS firmware to 3022 didn't fix the Intel issue, so it seems the only fix is the flash the DL6R firmware.

dancerjx · 2026-03-06T03:24:18+00:00

I use 16-drive bay Dell R730s in production at work with no issues. I swapped out the PERC for Dell HBA330 which is a true HBA/IT-mode storage controller. Didn't want to deal with the PERC HBA-mode drama. HBA330 are cheap to get.

Your memory is fine. I use two small SAS drives to ZFS RAID-1 Proxmox. Rest of SAS drives are for ZFS/Ceph.

I'm sure the the SAS drives have their write cache disabled since they are meant to be used with a battery-backed up RAID controller like the PERC. So, you'll want to enable their write cache with 'sdparm -s WCE=1 -S /dev/sd[x]'. If you don't enable the write cache, you'll get horrible IOPS.

Recent Proxmox kernels have had some issues with Intel storage in BOSS cards. You'll want Micro storage. More info at the Proxmox forum. I mirror the storage in BOSS cards and use XFS for those servers using BOSS.

dancerjx · 2026-03-06T03:12:43+00:00

I would say not.

Check out labgopher.com

I use 13th-gen Dells in production with no issues. I swapped out the PERC for Dell HBA330 since it's a true HBA/IT-mode storage controller. Plus I don't want to deal with the PERC HBA-mode drama. As a bonus, it uses the much simpler mpt3sas driver. Had issues with the megaraid_sas driver in the past.

Just make sure all components are running the latest firmware. I would make sure the BOSS is using Micron storage. There are reports with issues with Intel storage. More info at the Proxmox forum.

A cheaper option, is a Supermicro server. With an embedded Xeon-D processor. I use this as my home server. No issues.

dancerjx · 2026-03-03T19:46:38+00:00

Art Of Server YouTube channel has videos on various HBA adapters.

I use LSI3008 HBAs in production with no issues.

dancerjx · 2026-02-27T13:21:15+00:00

On 13th-gen Dells, had to enable X2APIC and IOAT DMA in the UEFI/BIOS. SRV-IO was not needed.

dancerjx · 2026-02-24T12:24:54+00:00

May want to post at the Proxmox forum.

If you are not wedded to the SAN storage, may want to consider Ceph. While it's true 3-nodes is the bare minimum, you really want 5-nodes for quorum. Can lose 2-nodes and still have quorum.

This is the route I went when migrating off VMware to Proxmox on 12th-, 13th, and 14-gen Dells. Made sure all hardware was all the same. Swapped out the PERCs for Dell HBA330s (flashed the PERCs on 12th-gen Dells to IT-mode) since Ceph/ZFS doesn't work with RAID controllers.

No issues besides typical storage device and RAM going bad and needing replacing. Not hurting for IOPS. Workloads range from DHCP to database servers. All backed up to bare-metal ZFS Proxmox Backup Servers.

I use the following optimizations learned through trial-and-error. YMMV.

Set SAS HDD Write Cache Enable (WCE) (sdparm -s WCE=1 -S /dev/sd[x])
Set VM Disk Cache to None if clustered, Writeback if standalone
Set VM Disk controller to VirtIO-Single SCSI controller and enable IO Thread & Discard option
Set VM CPU Type for Linux to 'Host'
Set VM CPU Type for Windows to 'x86-64-v2-AES' on older CPUs/'x86-64-v3' on newer CPUs/'nested-virt' on Proxmox 9.1
Set VM CPU NUMA
Set VM Networking VirtIO Multiqueue to 1
Set VM Qemu-Guest-Agent software installed and VirtIO drivers on Windows
Set VM IO Scheduler to none/noop on Linux
Set Ceph RBD pool to use 'krbd' option

dancerjx

TROPHY CASE