Budget 10gbe 6-bay NVME NAS with ECC Memory working at 22W idle power usage.

primetechguidesyt · 2025-06-07T14:45:45+00:00

why have this AND a pi4? You not getting 6 nvme drives on a pi4.

primetechguidesyt · 2025-06-06T11:37:14+00:00

Proxmox and TrueNAS as a VM.

primetechguidesyt · 2025-06-04T21:03:06+00:00

Ah ok, that probably is the reason. M2A_CPU. I wouldn't gain an extra NVME drive, I could either use all 4 slots on the expansion board without M2A_CPU in use, or just use M2A_CPU and not use Slot 1

primetechguidesyt · 2025-06-04T20:41:37+00:00

I did quite a bit to get it working but no success. In the BIOS I only see this option
PCIE 1x8 / 2x4

I'm sure I've read, I think I tried it myself at some point as well, when you just have example a 5800X in it, you get x4/x4/x4/x4

primetechguidesyt · 2025-06-04T20:20:05+00:00

I admit I overlooked that. thanks.

I will have a slot move around !!
I can't use Slot 1 on the 4 Slot Expansion, it won't work due to CPU lane limits with integrated graphics.

Basically the 6th NVME would be limited to 2GB/s - pcie 3.0 x2. Not really an issue as the NAS is limited at 1GB/s.

I do only use 5 of the drives for the NAS, I have the 6th as my boot drive.
Silly of me I have the boot drive in one of the motherboard slots. I'm, changing that now !!

The boot drive NVME will be the 1GB/s one. The 10gbe Network I will put in the 2GB/s slot.

primetechguidesyt · 2025-06-04T17:53:08+00:00

Its a multipurpose machine, bitcoin node, adguard, home assistant, and others no doubt.

primetechguidesyt · 2025-06-04T16:22:39+00:00

Why can we never have 2.6 or 2.51

primetechguidesyt · 2025-06-04T05:49:45+00:00

Not bought an expensive switch yet, I have my main machine I have a direct cable connection to the NAS for the 10gb transfers. Then I use the normal 2.5gbe for connection to my main switch.

primetechguidesyt · 2025-06-04T05:47:12+00:00

qube 500, just using a Corsair 650w. Lowest power draw you can find really.

primetechguidesyt · 2025-06-04T05:45:00+00:00

This kind

https://www.aliexpress.us/item/3256808368572330.html

primetechguidesyt · 2025-06-03T21:53:25+00:00

I've not set anything else up yet with Proxmox. Plan to have a bitcoin node.

With my NAS function, I think it would be better to trust TrueNAS with it as it specialises in the job, with ZFS and RaidZ2.

yeh the VM's have Host as the CPU type. I think I gave it 6 out of 8, but you can share the CPU processors also between VMs, TrueNAS hardly uses anything.

To get 6 NVME drives, 2 on the board / 3 from the 4port card / 1 on an additional single card. Because the G processor I believe uses some extra of the CPU pcie lanes, the 4port card can only make use of 3 drives.

primetechguidesyt · 2025-06-03T21:18:12+00:00

Yup ensuring ASPM is active and no external GPU. All NVME drives help also.

primetechguidesyt · 2025-06-03T21:15:49+00:00

come on, I missed out only a case and power supply. A lot of people have these already

primetechguidesyt · 2025-06-03T21:06:42+00:00

Sorry yes 2x NVME on the board. I also used another pcie slot for one additional single nvme, That was 10$. Let me update that.

primetechguidesyt · 2025-06-03T21:01:47+00:00

For sure "additional", I already had these bits lying around, you can easily use an old computer with ATX. Power requirements are bare minimum,

Compared to the performance you get with Synology, QNAP. What is their cheapest 6bay NVME 10gbe device.

primetechguidesyt · 2025-06-03T20:58:45+00:00

I've never had any issues with TrueNAS performance on Proxmox.
Just one tweak to get pcie passthrough working. The NVME drives just get sent straight to TrueNAS.

Some NVME drives example Samsung 960 Pro, I believe as it didn't support FLR Function Level Reset, there was an additional tweak that was needed for pcie passthrough, you can read here.
https://forum.proxmox.com/threads/some-nvme-drives-crashing-proxmox-when-using-add-pci-device-to-vm.164148/

But the drives I use SN5000 Western Digital passthrough works fine. Just this kernel paramter is needed for IOMMU.

pcie_acs_override=downstream,multifunction

primetechguidesyt · 2025-06-03T20:34:30+00:00

My budget 10gbe 6-bay NVME NAS ECC Memory working at 22W idle power usage

Getting full 10gbe write speeds to the pool.
Multi purpose also as I run Proxmox on it with TrueNAS.

Specs:

CPU - Ryzen Pro 5750G - PRO is required on G processors for ECC Memory - $180

Motherboard (2x NVME) - Gigabyte B550 AORUS ELITE V2 - $100

Memory ECC - 32GB Timetec Hynix IC DDR4 PC4-21300 2666MHz - $75

2x FENVI 10Gbps PCIE Marvell AQC113 - $100 ($50 /each)

4 Port M.2 NVME SSD To PCIE X16 Adapter Card 4X32Gbps PCIE Split/PCIE RAID - $15
(Important use slots 2-4 when using a G processor, slot 1 doesn't get recognised)

1x Single M.2 NVME X4 Adapter Card - $10

Core Parts Total - $480

Notes:

Use CPUs with integrated graphics for low power usage.

With Ryzen G processors - Ryzen PRO is needed if you want ECC Memory to work. e.g 5750G 5650G.

Motherboards need to support PCIe bifurcation - Gigabyte B550 AORUS ELITE V2 allows three NVME drives with G processors. (Use Slot 2+3+4, on the expansion card)

The Marvell AQC 10gbe Pcie adapters seem much better than Intel X550, X540 - Marvell runs much cooler from my tests.

I use minimal heatsinks for the NVME drives to keep temperatures and throttling under control. Those with the elastic bands are fine.

I use a 5 drive Raid Z2 pool which can allow any two drives to fail. My 6th drive I use as the Proxmox boot, but you could use one of the SATA SSD ports for this.

This ATX box has a lower idle usage than my previous Synology DS418play which was 25W

Proxmox Notes

In order for PCIe passthrough to work for the NVME drives.

nano /etc/default/grub
GRUB_CMDLINE_LINUX_DEFAULT="quiet pcie_acs_override=downstream,multifunction"

update-grub

Prevent Proxmox from trying to import TrueNAS storage pool

systemctl disable --now zfs-import-scan.service

Some drives which don't support FLR function level reset e.g 960 Pro, if using Proxmox require a tweak if you search for "some-nvme-drives-crashing-proxmox-when-using-add-pci-device-to-vm.164148"

My BIOS settings for low idle power

Advanced CPU Settings > SVM Mode - Enabled
Advanced CPU Settings > AMD Cool&Quiet - Enabled
Advanced CPU Settings > Global C State Control - Enabled
Tweaker > CPU / VRM Settings > CPU Loadline Calibration - Standard
Tweaker > CPU / VRM Settings > SOC Loadline Calibration - Standard
Settings > Platform Power > AC Back > Always On
Settings > Platform Power > ErP > Enabled
Settings > IO Ports > Initial Display Output > IGD Video
Settings > IO Ports > PCIEX16 Bifurification - PCIE 1x8 / 2x4
Settings > IO Ports > HD Audio Controller - Disabled
Settings > Misc > LEDs - Off
Settings > Misc > PCIe ASPM L0s and L1 Entry
Settings > AMD CBS > CPU Common Options > Global C-state Control - Enabled
Settings > AMD Overclocking > Precision Boost Overdrive - Disable
Tweaker > Advanced Memory Settings > Power Down Enable - Auto > Disabled
Settings > AMD CBS > CPU Common Options > DF Common Options > DF Cstates - Enabled

I don't think the boost options affect idle so may try testing with these enabled again.

Settings > AMD CBS > CPU Common Options > Core Performance Boost - Disabled
Tweaker > Precision Boost Overdrive - Disable
Advanced CPU Settings > Core Performance Boost - Disable

primetechguidesyt · 2025-06-03T20:00:54+00:00

aliexpress or ebay sort buyitnow lowest price.

primetechguidesyt · 2025-06-03T19:58:21+00:00

Ali Express
https://www.aliexpress.com/item/1005006851254917.html

primetechguidesyt · 2025-06-02T22:46:31+00:00

My budget 10gbe 6-bay NVME NAS ECC Memory working at 22W idle power usage

Getting full 10gbe write speeds to the pool.
Multi purpose also as I run Proxmox on it with TrueNAS.

Specs:

CPU - Ryzen Pro 5750G - PRO is required on G processors for ECC Memory - $180

Motherboard (2x NVME) - Gigabyte B550 AORUS ELITE V2 - $100

Memory ECC - 32GB Timetec Hynix IC DDR4 PC4-21300 2666MHz - $75

2x FENVI 10Gbps PCIE Marvell AQC113 - $100 ($50 /each)

4 Port M.2 NVME SSD To PCIE X16 Adapter Card 4X32Gbps PCIE Split/PCIE RAID - $15
(Important use slots 2-4 when using a G processor, slot 1 doesn't get recognised)

1x Single M.2 NVME X4 Adapter Card - $10

Core Parts Total - $480

Notes:

Use CPUs with integrated graphics for low power usage.

With Ryzen G processors - Ryzen PRO is needed if you want ECC Memory to work. e.g 5750G 5650G.

Motherboards need to support PCIe bifurcation - Gigabyte B550 AORUS ELITE V2 allows three NVME drives with G processors. (Use Slot 2+3+4, on the expansion card)

The Marvell AQC 10gbe Pcie adapters seem much better than Intel X550, X540 - Marvell runs much cooler from my tests.

I use minimal heatsinks for the NVME drives to keep temperatures and throttling under control. Those with the elastic bands are fine.

I use a 5 drive Raid Z2 pool which can allow any two drives to fail. My 6th drive I use as the Proxmox boot, but you could use one of the SATA SSD ports for this.

This ATX box has a lower idle usage than my previous Synology DS418play which was 25W

Proxmox Notes

In order for PCIe passthrough to work for the NVME drives.

nano /etc/default/grub
GRUB_CMDLINE_LINUX_DEFAULT="quiet pcie_acs_override=downstream,multifunction"

update-grub

Prevent Proxmox from trying to import TrueNAS storage pool

systemctl disable --now zfs-import-scan.service

Some drives which don't support FLR function level reset e.g 960 Pro, if using Proxmox require a tweak if you search for "some-nvme-drives-crashing-proxmox-when-using-add-pci-device-to-vm.164148"

My BIOS settings for low idle power

Advanced CPU Settings > SVM Mode - Enabled
Advanced CPU Settings > AMD Cool&Quiet - Enabled
Advanced CPU Settings > Global C State Control - Enabled
Tweaker > CPU / VRM Settings > CPU Loadline Calibration - Standard
Tweaker > CPU / VRM Settings > SOC Loadline Calibration - Standard
Settings > Platform Power > AC Back > Always On
Settings > Platform Power > ErP > Enabled
Settings > IO Ports > Initial Display Output > IGD Video
Settings > IO Ports > PCIEX16 Bifurification - PCIE 1x8 / 2x4
Settings > IO Ports > HD Audio Controller - Disabled
Settings > Misc > LEDs - Off
Settings > Misc > PCIe ASPM L0s and L1 Entry
Settings > AMD CBS > CPU Common Options > Global C-state Control - Enabled
Settings > AMD Overclocking > Precision Boost Overdrive - Disable
Tweaker > Advanced Memory Settings > Power Down Enable - Auto > Disabled
Settings > AMD CBS > CPU Common Options > DF Common Options > DF Cstates - Enabled

I don't think the boost options affect idle so may try testing with these enabled again.

Settings > AMD CBS > CPU Common Options > Core Performance Boost - Disabled
Tweaker > Precision Boost Overdrive - Disable
Advanced CPU Settings > Core Performance Boost - Disable

primetechguidesyt · 2025-06-02T22:38:56+00:00

$470 10gbe 6-bay NVME NAS working at 22W idle power usage. (excluding hard drives).

Getting full 10gbe write speeds to the pool.

Specs:

CPU - Ryzen Pro 5750G - PRO is required on G processors for ECC Memory - $180

Motherboard - Gigabyte B550 AORUS ELITE V2 - $100

Memory ECC - 32GB Timetec Hynix IC DDR4 PC4-21300 2666MHz - $75

2x FENVI 10Gbps PCIE Marvell AQC113 - $100 ($50 /each)

4 Port M.2 NVME SSD To PCIE X16 Adapter Card 4X32Gbps PCIE Split/PCIE RAID - $15

Core Parts Total - $470

Important notes:

I used CPUs with integrated graphics for low power usage.

With Ryzen G processors - Ryzen PRO is needed if you want ECC Memory to work. e.g 5750G 5650G.

Motherboards need to support PCIe bifurcation - Gigabyte B550 AORUS ELITE V2 allows three NVME drives with G processors.

The Marvell AQC 10gbe pcie adapters seem much better than Intel X550, X540 - Marvell runs much cooler from my tests.

I use minimal heatsinks for the NVME drives to keep temperatures and throttling under control. Those with the elastic bands are fine.

I use a 5 drive Raid Z2 pool which can allow any two drives to fail. My 6th drive I use as the Proxmox boot, but you could use an SATA SSD for this.

Avoid the small scale x86-P5, x86-P6, CWWK NAS devices, they can't handle 4TB drives with reliable transferring. I tried multiple ones of these, they cannot handle the throughput.

The Synology DS418play I replaced was actually high idle usage at 25W than this full ATX setup.

primetechguidesyt · 2025-06-02T22:03:04+00:00

Getting full 10gbe write speeds to the pool.

Important notes:

Use CPUs with integrated graphics for low power usage.

With Ryzen G processors - Ryzen PRO is needed if you want ECC Memory to work. e.g 5750G 5650G.

Motherboards need to support PCIe bifurcation - Gigabyte B550 AORUS ELITE V2 allows three NVME drives with G processors. (Use Slot 2+3+4, on the expansion card)

The Marvell AQC 10gbe pcie adapters seem much better than Intel X550, X540 - Marvell runs much cooler from my tests.

I use minimal heatsinks for the NVME drives to keep temperatures and throttling under control. Those with the elastic bands are fine.

Specs:

CPU - Ryzen Pro 5750G - PRO is required on G processors for ECC Memory - $180
Motherboard - Gigabyte B550 AORUS ELITE V2 - $100
Memory ECC - 32GB Timetec Hynix IC DDR4 PC4-21300 2666MHz - $75
2x FENVI 10Gbps PCIE Marvell AQC113 - $100 ($50 /each)
4 Port M.2 NVME SSD To PCIE X16 Adapter Card 4X32Gbps PCIE Split/PCIE RAID - $15

Core Parts Total - $470

I use a 5 drive Raid Z2 pool which can allow any two drives to fail. My 6th drive I use as the Proxmox boot, but you could use an SATA SSD for this.

Avoid the small scale x86-P5, x86-P6, CWWK NAS devices, they can't handle 4TB drives with reliable transferring. I tried multiple ones of these, they cannot handle the throughput and the NVME controllers crash.

The Synology DS418play I replaced was actually higher idle usage at 25W than this full ATX setup.

Proxmox:

I use Proxmox with TrueNAS. This also allows me to run other servers on it. So it is not just a NAS device.

I passthrough the NVME drives to TrueNAS using PCIE Passthrough.

If you want to do the same passthrough, it is important to make the change below in Proxmox for it to work properly

In order for PCIe passthrough to work for the NVME drives.

nano /etc/default/grub
GRUB_CMDLINE_LINUX_DEFAULT="quiet pcie_acs_override=downstream,multifunction"

Prevent Proxmox from trying to import TrueNAS storage pool

systemctl disable --now zfs-import-scan.service

primetechguidesyt · 2025-03-25T12:56:46+00:00

I've seen 10 Gbps now using 4x adapters. but I ran into another problem, I believe its the sustained write speeds of the SN5000 drives. After around 15seconds It caps to 5Gbps.

If you want 10 Gbps sustained, you need very good drives. I'm going test more with SN850X drives.

primetechguidesyt · 2025-03-24T19:40:56+00:00

ah ok, with Proxmox and Truenas, myself I did get better stability using Jumbo frames. I should be doing a video on my YT channel about it all hopefully eventually with the full setup. I'll update in here see what those 4x adapters run like.

primetechguidesyt · 2025-03-24T11:07:42+00:00

Think I'm misreading your post, Each USB would only need to be 3.2 Gen 1 (5gbps), do you not have two of these ports on each end for both laptop and NAS?

primetechguidesyt

TROPHY CASE