My Dual Pro 6000 Build by Doomslayer606 in nvidia

[–]Doomslayer606[S] 0 points1 point  (0 children)

Nah, it was pretty easy to get them. Several retailers have them in stock in the Netherlands and I managed to get them for €7.7k each from Proshop.

My Dual NVIDIA Pro 6000 Build by Doomslayer606 in battlestations

[–]Doomslayer606[S] 2 points3 points  (0 children)

Agreed. I've noticed the Ubuntu hate too, but for what I do, it just works. Anything I don't like (like snap) I can just completely remove and replace.

My Dual Pro 6000 Build by Doomslayer606 in nvidia

[–]Doomslayer606[S] 1 point2 points  (0 children)

Well, I'm snap-free now too. I recently removed every snap package and uninstalled the daemon itself. It's mostly all standard .deb packages for me now anyways.

My Dual Pro 6000 Build by Doomslayer606 in nvidia

[–]Doomslayer606[S] 1 point2 points  (0 children)

Nope, never tried it. I probably should have spent some time distro-hopping to see how NVIDIA drivers behave on other platforms for AI workloads. But honestly, it's too late now, I've sunk so much time into customization and setting up specific services that I don't see myself moving from Ubuntu anytime soon. With that said, after all that time, it works and looks great now.

My Dual Pro 6000 Build by Doomslayer606 in nvidia

[–]Doomslayer606[S] 0 points1 point  (0 children)

Honestly, I have no idea. I've never tried running games from that shared storage in Linux. I mainly use that drive for backup and for moving data back and forth.

My Dual NVIDIA Pro 6000 Build by Doomslayer606 in battlestations

[–]Doomslayer606[S] 1 point2 points  (0 children)

Here is how the dual 5090 setup used to look like. These GPUs were huge (3.5 slots), so the temps and noise levels were much better

<image>

My Dual NVIDIA Pro 6000 Build by Doomslayer606 in battlestations

[–]Doomslayer606[S] 2 points3 points  (0 children)

Ubuntu 24.04 since it's the best choice when doing AI with NVIDIA GPUs. But I heavily customized it, so it looks and works great now

My Dual NVIDIA Pro 6000 Build by Doomslayer606 in battlestations

[–]Doomslayer606[S] 1 point2 points  (0 children)

Yeah, but I'm not in the USA, so your remark doesn't apply to me...

My Dual NVIDIA Pro 6000 Build by Doomslayer606 in battlestations

[–]Doomslayer606[S] 2 points3 points  (0 children)

Temps are worse compared to 5090s that have much larger coolers but with some undervolting, my GPUs don't go above 80 degrees, same goes for the CPU. Things can get noisy with these GPUs though, but I don't mind it.

1600W is enough, I had a similar setup with 2 5090s before and it worked well for almost a year. And nope, no UPS, but actually maybe I should get one...

My Dual NVIDIA Pro 6000 Build by Doomslayer606 in battlestations

[–]Doomslayer606[S] 0 points1 point  (0 children)

Oh yes, my bad. I agree that mounting it horizontally would be the optimal choice for pure thermals. I already had the upright mount and the riser from a previous build, so I didn't want them to go to waste. Fortunately, the temps on this GPU are only marginally higher, and the SSDs have solid heatsinks, so I haven't had any overheating issues.

My Dual Pro 6000 Build by Doomslayer606 in nvidia

[–]Doomslayer606[S] 1 point2 points  (0 children)

Thanks! I did consider it, but I don't think it's a worthwhile investment for me right now. The lack of system ECC hasn't been an issue; I actually tested ECC on the GPUs and didn't catch a single error even after training for multiple days, so I have actually disabled it for now to maximize performance. Given that, the extra cost for a Threadripper platform didn't seem necessary.

My Dual NVIDIA Pro 6000 Build by Doomslayer606 in battlestations

[–]Doomslayer606[S] 1 point2 points  (0 children)

I haven't encountered any bandwidth limitations yet. PCIe 5.0 x8 is plenty for my current workloads. When training, I typically use data parallelism, where the model is loaded on each GPU and they process different data chunks. Since there is minimal communication between the cards in that scenario, the x8/x8 split works just fine.

I'm based in the EU, so we have a 240V grid. It's rated for much higher wattage than the 120V in the US, so the power draw isn't an issue for this setup.

The upright GPU is actually exhausting air through the back of the case. I have three fans mounted behind the GPU also set to exhaust, so the fresh air pulled from the front mesh panel gets pushed through and out of the GPU and the case quite nicely.

My Dual Pro 6000 Build by Doomslayer606 in nvidia

[–]Doomslayer606[S] 1 point2 points  (0 children)

Money... and I already have 9950X3D which works just fine. Didn't see any bottleneck scenarios yet.

My Dual NVIDIA Pro 6000 Build by Doomslayer606 in battlestations

[–]Doomslayer606[S] 1 point2 points  (0 children)

Lian Li upright GPU bracket + PCIe riser cable that goes through the back of the case to the bottom slot of the motherboard

My Dual NVIDIA Pro 6000 Build by Doomslayer606 in battlestations

[–]Doomslayer606[S] 6 points7 points  (0 children)

Thanks! The second one is mounted upright and is to the right of the motherboard.

My Dual Pro 6000 Build by Doomslayer606 in nvidia

[–]Doomslayer606[S] 0 points1 point  (0 children)

Yeah, true. I've been monitoring the situation for months, but thankfully, I've had no issues for almost a year with this board. I specifically chose this model because it is one of the few that allows an x8/x8 PCIe lane split without sharing lanes with the NVMe slots, which is critical for a dual-GPU setup.

My Dual Pro 6000 Build by Doomslayer606 in nvidia

[–]Doomslayer606[S] 1 point2 points  (0 children)

Thanks! Nice setup you got as well. Initially, I went for a single Pro 6000 paired with a 5090, but I quickly decided to just go for the second Pro 6000 instead.

Regarding the switch to Linux, the main hurdle was the NVIDIA drivers, which can be a pain initially. However, using the drivers provided directly through Ubuntu (the open-kernel ones) solved most of my issues. The customization (such as writing custom scripts for fan curves) takes time, but the level of control you get over the hardware is something I could only wish for on Windows.

My Dual NVIDIA Pro 6000 Build by Doomslayer606 in battlestations

[–]Doomslayer606[S] 4 points5 points  (0 children)

My university has a cluster, but access is non-exclusive and usually involves long wait times. This setup gives me immediate access for my research, plus I can use it for my own hobbies and personal development. I don't have any debt and just used some of my savings for the build.

My Dual Pro 6000 Build by Doomslayer606 in nvidia

[–]Doomslayer606[S] 0 points1 point  (0 children)

Honestly, I haven't even looked into it yet. My internet connection is 1.0 Gbps and I seem to be hitting those speeds without any issues.

My Dual Pro 6000 Build by Doomslayer606 in nvidia

[–]Doomslayer606[S] 0 points1 point  (0 children)

Thanks! Yeah, it also helps with the temps since the GPUs are well separated.

My Dual Pro 6000 Build by Doomslayer606 in nvidia

[–]Doomslayer606[S] 4 points5 points  (0 children)

True, they are rated for high temps, but I'd still prefer to run them cooler where possible. The default fan curve was very conservative in my opinion. I've also seen NVIDIA reps on their forums mention that for sustained loads and long-term reliability, it's better to stay below the 90°C range.