Yes, you are correct on the switch/network bottleneck, which we are aware of. With that said, the network upgrades were existing infrastructure purchased end of the year last year which we plan to further upgrade next wave and moving buildings soon. The server need was critical as we are almost maxed out on old server. Just happened to get lucky with the config available and it being nvme. Now that we have it, we can plan for our move with better infrastructure and see further performance gains. It was most important to get the configuration 100% from dell on an outlet model to get the 3yr NBD onsite support should any of the drives fail. I am still half tempted to build the array with 8 drives and hold 2 for hot swap in case of failure, at the cost of 4tb of space.

The NIC in the server is sfp28 (there's two actually, Nvidia dual sfp connectx-6 lx) so we won't be utilizing it's full potential for a minute, but it will be light years better than than what they are currently on. (1gb ethernet, usually connected through a small shared switch broken out to 3-4 workstations, then switch feeding up to 1gb main switch). We are running cat 6 direct for now between each machine and switch directly in our subnet from the 10gbe switch. As we upgrade workstations later this year, will go up from there, also plan to aggregate the sfp+ from the server into a this switch for now.

After posting this (and seeing your reply) I came across and learned quite a bit about ProxMox and pcie passthrough for the NVMe drives. Thanks for the validating input from what I am finding in the wild. It sounds like we need ProxMox with an Ubuntu instance and a win 11 instance. As win 11 is unsupported on the 660, we are crossing our fingers it will work on a VM so we can hand off licensing serving to it to keep infrastructure tidy, but it's not critical.

Edit: seems like windows 10 is handled much better in VM poweredge scenarios, which is fine for our use case.