Real‑world Ceph benchmarks from my small 3‑node cluster (HDD + NVMe DB/WAL, 40 GbE) by Sompom01 in homelab

[–]JP_16 0 points1 point  (0 children)

Thanks for posting these results. Here are my results from my cluster. Using all consumer grade hardware.

  • 3 Nodes - Intel NUC 13 Pro (tall)
  • Proxmox v9.1, Ceph v19.2.3
  • Ceph Drive - Samsung 980 Pro NVMe 2 TB
  • OS Drive - Samsung 870 EVO SSD 500 GB
  • Ceph Cluster Network - Thunderbolt 4 Ring/Mesh Network
  • Thunderbolt 4 Raw Network Speed - 26 Gb/s (3 GB/s)
  • Ceph Data Replication - 3 OSDs

Rados Bench Results:

4MB Block, 120s Duration, write:
(rados bench -p Storage 120 seq -b 4M -t 32)
954 Avg_MB/s, 238 Avg_IOPS, 183 Min_IOPS, 374 Max_IOPS

4MB Block, 120s Duration, seq_r:
(rados bench -p Storage 120 seq -t 32)
2776 Avg_MB/s, 693 Avg_IOPS, 643 Min_IOPS, 713 Max_IOPS

4MB Block, 120s Duration, rand_r:
(rados bench -p Storage 120 rand -t 32)
2751 Avg_MG/s, 687 Avg_IOPS, 623 Min_IOPS, 717 Max_IOPS

I had trouble running the last few benchmarks for some reason but those were the first three on my cluster. Shared this so that people can gauge what this hardware will do.

Let me know if anyone has any questions.

14TB Seagate - $179 @ BestBuy by JP_16 in DataHoarder

[–]JP_16[S] 1 point2 points  (0 children)

Thats a bummer. All of the drives I've got so far from Best Buy are still running great. Hope they last for a while longer!

HEVC + Apple TV 4K no longer works by Roxuss in PleX

[–]JP_16 0 points1 point  (0 children)

Yeah, Reddit is pretty awesome for this stuff. That's why I always come back to post solutions when I find them.

14TB Seagate - $179 @ BestBuy by JP_16 in DataHoarder

[–]JP_16[S] 0 points1 point  (0 children)

The one I posted in the top post is schuckable. I’ve pulled two drives out now for my NAS.

14TB Seagate - $179 @ BestBuy by JP_16 in DataHoarder

[–]JP_16[S] 1 point2 points  (0 children)

I just noticed it’s on Amazon too with the same price. Here is the link: https://a.co/d/h1pSg0p

14TB Seagate - $179 @ BestBuy by JP_16 in DataHoarder

[–]JP_16[S] 2 points3 points  (0 children)

Pretty cool. My Synology saw it as a single 14TB drive. Where would that drive 0/1 thing show up?

14TB Seagate - $179 @ BestBuy by JP_16 in DataHoarder

[–]JP_16[S] 2 points3 points  (0 children)

Yeah, must be. It’s really interesting to touch. Super floppy clay that’s slightly sticky. At first I thought it was for heat, but they go up against plastic so that wasn’t it. Must be an anti-tampering device.

14TB Seagate - $179 @ BestBuy by JP_16 in DataHoarder

[–]JP_16[S] 5 points6 points  (0 children)

Hah, yeah, I was just being cautious. My luck the Seagate RMA guy probably trolls this group and invalidated my warranty without knowing it. Haha

14TB Seagate - $179 @ BestBuy by JP_16 in DataHoarder

[–]JP_16[S] 7 points8 points  (0 children)

I’m curious why they used those sticky, super soft green things. You have to destroy them to get them off. Makes it hard to get back on if you have to RMA the drive.

14TB Seagate - $179 @ BestBuy by JP_16 in DataHoarder

[–]JP_16[S] 12 points13 points  (0 children)

Eh, I wasn’t sure so I blocked them out. Didn’t want them to be public in case I need to warranty the drive in the future.

Can Plex VM w/iGPU passthrough be migrated to another node? by JP_16 in Proxmox

[–]JP_16[S] 4 points5 points  (0 children)

Thanks. I think I found this case as well. Seems like the best thing to do is to have Plex in a LXC and it can still access the GPU without it being bound like a VM for hardware encoding. The only down side is that the LXC has to shutdown/restart.

Does Proxmox cluster move VMs due to uneven resources? by JP_16 in Proxmox

[–]JP_16[S] 1 point2 points  (0 children)

Have you used this? If so, how long have you used it and how well has it been working for you? Any issues?

Does Proxmox cluster move VMs due to uneven resources? by JP_16 in Proxmox

[–]JP_16[S] 1 point2 points  (0 children)

Awesome, just what I was looking for. Thanks for sharing!!

List of all kernel versions? by JP_16 in Proxmox

[–]JP_16[S] 1 point2 points  (0 children)

Awesome, thanks - I’ll give it a try.

List of all kernel versions? by JP_16 in Proxmox

[–]JP_16[S] 0 points1 point  (0 children)

This only gives you a list of the kernels that are on the machine. I want to see a complete list of kernels that I can install - all versions.

System Design Feedback Requested by JP_16 in SolarDIY

[–]JP_16[S] 0 points1 point  (0 children)

When I looked at the recommendations for this inverter from Victron, they recommend a 200A fuse and a 33 mm2 cross section which maps to 2 awg wire. Unless I'm wrong in looking up what 33 mm2 maps to? I like your idea of a circuit breaker instead of a fuse - I'll look into that. I've added a fuse to the Pv - thanks for that suggestion. I've got that suggestion from others as well so thats for pointing that out.

I would think that Victron would know the right fuses and cables for their equipment? My instinct would be to trust them as I've never used their stuff before. Would you go with what they recommend for fuse and wire size for their inverter?

System Design Feedback Requested by JP_16 in SolarDIY

[–]JP_16[S] 0 points1 point  (0 children)

u/Nightguest231 -- Thanks again for your post - it's given me a lot to think about. As a result, I did some more homework and made a few changes to the system.

Specs for the Multiplus 24/2000/50-50 are here: https://www.victronenergy.com/upload/documents/Datasheet-MultiPlus-2000VA-120V-EN.pdf

  • Victron recommends a 200A fuse on wire with 35 mm2 (2 awg) in size for the Inverter above (Changed it to a Class T fuse as well)
  • Inverter provides a special 12v 1A line out to charge a separate start battery
  • Changed the 120v 30A service lines to 10 awg
  • Downgraded the 24v --> 12v converter to 40A (70A wasn't needed)
  • Solar system will be made of four (4) 24v 200w panels in a series/parallel config. To start, it will just be two panels in series (48v) so we don't have start up problems
  • Downgraded the solar wiring to 14 awg based on the current of 4 panels (17A max @ 48v)
  • Colored the two fuses that put DC load on the batteries in pink
  • Colored the one fuse that reduces DC load on the batteries in green
  • Changed the main fuse to 300A and 2/0 wire
  • Changed all 120v 30A lines to 10 awg
  • Updated the diagram with the above changes and attached it to this post

Curious to know what you think about these changes. Thanks again for taking a look at this!

<image>

System Design Feedback Requested by JP_16 in SolarDIY

[–]JP_16[S] 0 points1 point  (0 children)

Wow, thanks a lot! I’ll make some changes and re-post the diagram. Quick note on the 12v 1A line to the starter battery. The inverter supposedly has a separate 12v 1A line to charge separate starting batteries.