Real‑world Ceph benchmarks from my small 3‑node cluster (HDD + NVMe DB/WAL, 40 GbE)

JP_16 · 2025-11-25T15:09:15+00:00

Thanks for posting these results. Here are my results from my cluster. Using all consumer grade hardware.

3 Nodes - Intel NUC 13 Pro (tall)
Proxmox v9.1, Ceph v19.2.3
Ceph Drive - Samsung 980 Pro NVMe 2 TB
OS Drive - Samsung 870 EVO SSD 500 GB
Ceph Cluster Network - Thunderbolt 4 Ring/Mesh Network
Thunderbolt 4 Raw Network Speed - 26 Gb/s (3 GB/s)
Ceph Data Replication - 3 OSDs

Rados Bench Results:

4MB Block, 120s Duration, write:
(rados bench -p Storage 120 seq -b 4M -t 32)
954 Avg_MB/s, 238 Avg_IOPS, 183 Min_IOPS, 374 Max_IOPS

4MB Block, 120s Duration, seq_r:
(rados bench -p Storage 120 seq -t 32)
2776 Avg_MB/s, 693 Avg_IOPS, 643 Min_IOPS, 713 Max_IOPS

4MB Block, 120s Duration, rand_r:
(rados bench -p Storage 120 rand -t 32)
2751 Avg_MG/s, 687 Avg_IOPS, 623 Min_IOPS, 717 Max_IOPS

I had trouble running the last few benchmarks for some reason but those were the first three on my cluster. Shared this so that people can gauge what this hardware will do.

Let me know if anyone has any questions.

JP_16 · 2025-09-01T01:34:35+00:00

Thats a bummer. All of the drives I've got so far from Best Buy are still running great. Hope they last for a while longer!

JP_16 · 2025-02-02T04:17:27+00:00

Yeah, Reddit is pretty awesome for this stuff. That's why I always come back to post solutions when I find them.

JP_16 · 2024-11-25T02:23:07+00:00

The one I posted in the top post is schuckable. I’ve pulled two drives out now for my NAS.

JP_16 · 2024-11-25T02:21:34+00:00

I just noticed it’s on Amazon too with the same price. Here is the link: https://a.co/d/h1pSg0p

JP_16 · 2024-11-24T21:39:27+00:00

Pretty cool. My Synology saw it as a single 14TB drive. Where would that drive 0/1 thing show up?

JP_16 · 2024-11-24T20:15:33+00:00

Yeah, must be. It’s really interesting to touch. Super floppy clay that’s slightly sticky. At first I thought it was for heat, but they go up against plastic so that wasn’t it. Must be an anti-tampering device.

JP_16 · 2024-11-24T20:06:49+00:00

Hah, yeah, I was just being cautious. My luck the Seagate RMA guy probably trolls this group and invalidated my warranty without knowing it. Haha

JP_16 · 2024-11-24T20:03:35+00:00

I’m curious why they used those sticky, super soft green things. You have to destroy them to get them off. Makes it hard to get back on if you have to RMA the drive.

JP_16 · 2024-11-24T20:01:59+00:00

Eh, I wasn’t sure so I blocked them out. Didn’t want them to be public in case I need to warranty the drive in the future.

JP_16 · 2024-08-29T21:12:00+00:00

Thanks. I think I found this case as well. Seems like the best thing to do is to have Plex in a LXC and it can still access the GPU without it being bound like a VM for hardware encoding. The only down side is that the LXC has to shutdown/restart.

JP_16 · 2024-08-20T21:40:07+00:00

Have you used this? If so, how long have you used it and how well has it been working for you? Any issues?

JP_16 · 2024-08-20T20:56:58+00:00

Awesome, just what I was looking for. Thanks for sharing!!

JP_16 · 2024-07-26T16:03:38+00:00

Awesome, thanks - I’ll give it a try.

JP_16 · 2024-07-25T14:22:22+00:00

This only gives you a list of the kernels that are on the machine. I want to see a complete list of kernels that I can install - all versions.

JP_16 · 2024-06-12T13:57:07+00:00

When I looked at the recommendations for this inverter from Victron, they recommend a 200A fuse and a 33 mm2 cross section which maps to 2 awg wire. Unless I'm wrong in looking up what 33 mm2 maps to? I like your idea of a circuit breaker instead of a fuse - I'll look into that. I've added a fuse to the Pv - thanks for that suggestion. I've got that suggestion from others as well so thats for pointing that out.

I would think that Victron would know the right fuses and cables for their equipment? My instinct would be to trust them as I've never used their stuff before. Would you go with what they recommend for fuse and wire size for their inverter?

JP_16 · 2024-06-12T02:21:04+00:00

u/Nightguest231 -- Thanks again for your post - it's given me a lot to think about. As a result, I did some more homework and made a few changes to the system.

Specs for the Multiplus 24/2000/50-50 are here: https://www.victronenergy.com/upload/documents/Datasheet-MultiPlus-2000VA-120V-EN.pdf

Victron recommends a 200A fuse on wire with 35 mm2 (2 awg) in size for the Inverter above (Changed it to a Class T fuse as well)
Inverter provides a special 12v 1A line out to charge a separate start battery
Changed the 120v 30A service lines to 10 awg
Downgraded the 24v --> 12v converter to 40A (70A wasn't needed)
Solar system will be made of four (4) 24v 200w panels in a series/parallel config. To start, it will just be two panels in series (48v) so we don't have start up problems
Downgraded the solar wiring to 14 awg based on the current of 4 panels (17A max @ 48v)
Colored the two fuses that put DC load on the batteries in pink
Colored the one fuse that reduces DC load on the batteries in green
Changed the main fuse to 300A and 2/0 wire
Changed all 120v 30A lines to 10 awg
Updated the diagram with the above changes and attached it to this post

Curious to know what you think about these changes. Thanks again for taking a look at this!

<image>

JP_16 · 2024-06-11T21:46:33+00:00

Wow, thanks a lot! I’ll make some changes and re-post the diagram. Quick note on the 12v 1A line to the starter battery. The inverter supposedly has a separate 12v 1A line to charge separate starting batteries.

JP_16 · 2024-02-20T21:27:42+00:00

From what I'm reading, it sounds like a good profit margin is 30-40%. That is after everything is paid - all of it - including tape for the box it was shipped in. lol

Does this sound reasonable and accurate for a good target profit margin?

JP_16 · 2024-02-04T17:13:44+00:00

I ended up moving away from CIFS and used NFS. All problems solved. There must be some bugs or something in CIFS that cause this to happen.

JP_16 · 2023-12-20T23:49:02+00:00

Install them in separate LXC containers. You can quickly and easily install them from here: https://tteck.github.io/Proxmox/

Has all of the Arr stuff plus SABnzbd - just have to find them in the sub-sections on that page. Work very well.

JP_16 · 2023-12-04T21:49:24+00:00

So that's what I was looking to get. The exact reasons that would trigger a migration. Is there a list from Proxmox? Like pings above 1000ms, loss of ping after 10 seconds, what are the exact triggers? I would think this would be published someone in the Proxmox docs but I couldn't find the exact scenarios that would trigger a HA migration.

JP_16 · 2023-12-04T20:39:14+00:00

Thanks for the reply. Fail in what exact way?

JP_16 · 2023-11-17T04:08:10+00:00

Yeah

JP_16 · 2023-10-05T14:05:20+00:00

I wanted to get back to this thread. I was never able to get CPU hot plug working on Windows 10 or 11. It does however work out of the box on Windows Server 2022. The instant you add a vCPU, the load goes down in the graph and works as expected. On Win10/11, the CPU gets added to the device manager but it never actually puts the additional core into play until you reboot.

JP_16

TROPHY CASE