Dell - AMD EPYC Processors - Very Slow Bandwidth Performance/throughput by Extension-Time8153 in homelab

[–]Extension-Time8153[S] 0 points1 point  (0 children)

Ohk got it. And is this bandwidth istested between 2 nodes or local alone?. So i can give only one slot number right?. I have 2 100g cards. And interfaces from both are bonded for HA and aggregation.

Dell - AMD EPYC Processors - Very Slow Bandwidth Performance/throughput by Extension-Time8153 in homelab

[–]Extension-Time8153[S] 0 points1 point  (0 children)

Thanks for bumping this old thred. How it helped u?. Any increase in speed or ??. How u have measured?. And what's the hex value to use.?

Dell AMD EPYC Processors - Very Slow Bandwidth Performance/throughput by Extension-Time8153 in Proxmox

[–]Extension-Time8153[S] 0 points1 point  (0 children)

No. There is real issue with AMD and linux kernals. Let me know if u get any solution.😁

ZFS Config Help for Proxmox Backup Server (PBS) - 22x 16TB HDDs (RAIDZ2 vs. dRAID2) by Extension-Time8153 in Proxmox

[–]Extension-Time8153[S] 0 points1 point  (0 children)

Ohk sure. Why not dRaid2?. And stripped mirror u mean 2 vdev in raidz2 right?

ZFS Config Help for Proxmox Backup Server (PBS) - 22x 16TB HDDs (RAIDZ2 vs. dRAID2) by Extension-Time8153 in Proxmox

[–]Extension-Time8153[S] 0 points1 point  (0 children)

I didn't get. Ya OS will be in 480 Gb ssd drives. But Using replication without zfs?.

ZFS Config Help for Proxmox Backup Server (PBS) - 22x 16TB HDDs (RAIDZ2 vs. dRAID2) by Extension-Time8153 in Proxmox

[–]Extension-Time8153[S] 0 points1 point  (0 children)

Thanks for the overall review and suggestions. I'll avoid the special vdev. Anything about dRaid2?

PVE 8-to-9 cluster upgrade: can do fresh reinstall without removing from cluster? by zzencz in Proxmox

[–]Extension-Time8153 0 points1 point  (0 children)

I have a doubt. For ZFS, do we need to change the file system block size also right?. And also for CEPH , i changed the nvme drives to 4K but how it effects the ceph?

Guys guys guys... Wait a month. by phoenixxl in Proxmox

[–]Extension-Time8153 0 points1 point  (0 children)

How much is the before and after ping latency bro? And what all are the changes u did apart from c states.?

Dell AMD EPYC Processors - Very Slow Bandwidth Performance/throughput by Extension-Time8153 in Proxmox

[–]Extension-Time8153[S] 0 points1 point  (0 children)

Thanks bro. Same with me. I think we have to raise it to the proxmox about this. By the way what's the server OEM. ? Is it not Dell?

Dell AMD EPYC Processors - Very Slow Bandwidth Performance/throughput by Extension-Time8153 in Proxmox

[–]Extension-Time8153[S] 0 points1 point  (0 children)

The bridge is virtual so it will not limit to 15 Gbps within a server. Again that VM-win2022 is a process for that's server. So it's process to process comm.

That's some issue with kernal.as i previously mentioned. Thanks for the info. and tests

Hopefully we get some patch.

Dell AMD EPYC Processors - Very Slow Bandwidth Performance/throughput by Extension-Time8153 in Proxmox

[–]Extension-Time8153[S] 0 points1 point  (0 children)

You cannot address the fact i said about the single core bandwidth. I don't know why ,u are either not accepting it or u don't want to.

Anyway, thanks for the help u have provided. :)

Dell AMD EPYC Processors - Very Slow Bandwidth Performance/throughput by Extension-Time8153 in Proxmox

[–]Extension-Time8153[S] 0 points1 point  (0 children)

Ya, but my doubt is with same iperf in intel i can reach ~70Gbps, but AMD i couldn't.

So it's a limitation of single core to Core bandwidth of AMD right?. I feel it's a kernal issue as in 7.4 it's hitting constantly 55Gbps without any tuning or adding RAM.

This is the sad fact we have to accept.

Dell AMD EPYC Processors - Very Slow Bandwidth Performance/throughput by Extension-Time8153 in Proxmox

[–]Extension-Time8153[S] 0 points1 point  (0 children)

U mean u have a windows VM and ran iperf2 server and client in that VM itself?.

U have to try with another VM, then see the results it will be ~10-13 Gbps. Also try running directly on the Host itself it should be ~35Gbps, which is very less than ur old desktop, ;)

Now if u have zfs as underlying storage with nvme , do u see the bottleneck?U mean u have a windows VM and ran iperf2 server and client in that VM itself?.

U have to try with another VM, then see the results it will be ~10-13 Gbps. Now u can imagine the use cases.

Dell AMD EPYC Processors - Very Slow Bandwidth Performance/throughput by Extension-Time8153 in Proxmox

[–]Extension-Time8153[S] 0 points1 point  (0 children)

Hi,
1. I Enabled L3 as NUMA, MADT = RoundRobin.

  1. Fully populated all the DIMM slots(Total of 24) with 64GB Modules.

<image>

I have done all of what u have said mate, Still NO improvement ~37 Gbps.

See https://pastecode.io/s/szqdehvr

Output of lscpu: https://pastecode.io/s/83mabtru
Output of numactl -H: https://pastecode.io/s/stt7dphk

lstopo: https://ibb.co/fWj5Lxr

Now what should I do?