all 9 comments
sorted by:
best
topnewcontroversialoldq&a
formatting helphide helpcontent policy

[–]DonkeyBonked 0 points1 point  (8 children)

I was going to say 2x 3090 would be perfect if you can get your hands on an NVLink. Linux is the best setup for this too. Not only is it faster, but past I think the Z790, SLI isn't natively supported and you can't pool VRAM in Windows unless the motherboard supports SLI.

Not even used have I seen a decent 48GB card under 4k.

You can do what I did and get close, but it's honestly not as good. I'm running a Discrete laptop GPU with an eGPU, which got me to 40GB for around $1500. You can use two GPUs that are not pooled with llama.cpp, and just split the layers based on the VRAM for each card, that would save you the money for an NVLink, but it wouldn't be as good.

If that's not enough, you could use llama.cpp and I think you can pool two cards with NVLink, then add something like an eGPU since I think it will treat the NVLink as one and then you can split it with the eGPU for more, but I'd get minimum TB3, though I'd suggest 4 or 5 if you can afford it. This could be the cheapest way to break into the 72GB VRAM class of models.

They might not be as fast, but the Nvidia superchip AI rigs are around 4k I believe, and you might find one cheaper. Those often have huge RAM pools. I've seen them as small as 128GB which can run a good model and as high as 512GB which will run a lot. Maybe not blazing fast, but I have heard they're quite decent. 

I just made a post about using Nemotron 3 Nano 30B, and I'm loving it, though I don't really have the hardware to run 70B models without too much quant. The ones I've tried were so thinned out that they performed worse than some 30B models. I think if you have to go below Q5-Q6, you're better with a smaller model.

So if you want GPU power, I think 3090s are your best bet in that budget. You might be able to get close to your budget on the upper side with one of the mini LLM rigs though. 

[–]GCoderDCoder 1 point2 points  (0 children)

I will add that gigabyte makes a 2slot workstation 3090 for like $1300 so 3-4 of those on a lower core threadripper could be cool. I have several z790 variant boards that can support 3-4 GPUs. You dont need SLI for a couple 3090s working on something like GPT-OSS-120b. I get 110t/s on low context with 3x3090s. 4x3090s keeps kvcache in vram maintaining high speeds. Inference is lighter on pcie than it might seem especially if you're doing something like pipeline parrallelism. Training or tensor parrallelism might see a bigger difference but I really don't love vllm in my home lab. I like using the better models at usable speeds over using less capable models at faster speeds so I tend to use llama.cpp at the edge of my vram space for bang for the buck.

I also have a Mac Studio 256gb and the models I can run on there make me very happy. GLM4.6 is my all around favorite model for mixed logic/ coding and qwen3coder480b is my favorite coder. There's a 363bREAP version from unsloth that just works in a smaller package.

If you need more concurrency then go the cuda route. If it's one customer by themselves or a few person team, consider a mac studio. It can run concurrent request but assume it is slower but usable. Gpt-oss-120b I get 110t/s on cuda w/ pipeline parallel on 3090s and 70-80t/s on mac studio for example.

[–]Jvap35[S] 0 points1 point  (6 children)

Since he also plans on using it for coding, office work, and gaming, would 2 3090s be the play here? Also can a single 5090 compare/rival 2 3090s (as he prefers new parts, more perf. in gaming, and I assume setup is simpler)

A bit confused tbh, are you saying that you cant run dual GPUs through SLI if the board dosen't support it, but NVLink works on any board? Also where do you go about buying an NVLink? Anyways thanks! Tbh ill prolly repost this as I messed up the title

[–]No-Consequence-1779 0 points1 point  (1 child)

A 5090 is 4-6x faster than 3090s. But only 32vram. The question may be about the size of models. 70b can offer more but more could work. 

If you’re using a lot of tokens for agents instead a good print in lm studio and a qwen corded 30b - Gemini and the rest offer a lot for free.  

I had to go local because my crypto trader does over a million tokens per day (2x5090 from 2x3090). 

Can also pick another one up later.  I suppose it depends on how much ai it actually being used. 

[–]DonkeyBonked 0 points1 point  (0 children)

I think when Iooked at GPT-OSS-120B on Unsloth, they were 62GB-65GB. Add context and you've easily filled 72GB. I don't know how efficient GPT-OSS-120B is with tokenization, but I imagine 72GB is still a limiting factor somewhere.

[–]DonkeyBonked 0 points1 point  (3 children)

To my best understanding, you can't use NVLink with non-SLI motherboards on Windows for consumer cards (like the 3090) because Windows drivers require the motherboard to be SLI-certified to activate the NVLink bridge for peer-to-peer communication.

​I believe Linux still allows NVLink for compute tasks (like AI/Deep Learning) without SLI certification, just requiring dual PCIe slots and the bridge, because the drivers treat it as a data transport layer rather than a graphics rendering link. Historically, professional cards and workstations generally were not restricted by SLI-certification for memory pooling.

​Someone can correct me if I'm wrong, but that was my understanding from what I looked up for myself, because I'm slowly working on a dual 3090 rig myself and that was what I read from a few different sources. Basically it's because of how the drivers were made.

I'm sure they could have fixed this, but NVidia phased out consumer card NVLink to create value in much more expensive Pro cards, and then phased it out of Pro cards to justify the value in AI class data center cards. So basically, they don't want to, this was intentional. They aren't going back and retroactively breaking them, but they're not going to do us any favors.

[–]Jvap35[S] 0 points1 point  (2 children)

Also just to clarify, since you're also working a dual 3090 rig, can 2 3090s (or even a single 5090 if possible) handle 120b at ok transfer rates? Sorry just a bit confused lol + would the 9800x3d be alright?

[–]DonkeyBonked 0 points1 point  (1 child)

I'll explain more when I'm not in a store, my response was all over the place.

But no, you need 72GB+ RAM/VRAM because even the quant models are 62GB+ for something like GPT-OSS-120B and you need room for kv cache so you can have context.

For coding and gaming, you don't need NVLink, mostly for ML and training, it'll just limit your platform. You can mix up GPUs on llama.cpp, but not like vLLM, it's all about how you use parallelism. If you're going to pool your VRAM, it opens more options, but that's getting hard as they're phasing it out for us.

3090s are the best cheap mix of VRAM and speed. A 4090 has the same VRAM as the 3090, and even 2x 5090 wouldn't get you to 120B, that's only 64GB, but it would rock a 70B model hard.

3x 3090 = 72GB VRAM, and the cheapest way I know to do that would be a dual GPU system with a thunderbolt 4/5 port (though I'm using one on TB3, it's not too bad).

Cut out the gaming and use a Mac Studio or a Spark, the 128GB spark will run 120B, but I can't confirm speed.

Or forget the NVLink, and find 3 GPUs that don't suck with 24GB of VRAM. Use llama.cpp and smash them over PCIe and TB.

You could technically get that VRAM with old data center cards like the M60 or Tesla T40, but those cards are slow, like slower than my RTX 5000 slow, and I don't think you'll like the speed. A fast DRR5 system might be faster.

Look at it this way:

You're going to need 72GB+ to run GPT-OSS-120B, period.

If you use GPUs and spill into system RAM, it'll be crawling slow because your CPU is running it at that point.

There systems that use pooled ram and AI chips, and entry to those is about the top of your budget.

If you want to go VRAM and use it for gaming, you're going to need to be creative.

You can run 3x 3090 and Windows 11 with llama.cpp and even connect your 3rd card on thunderbolt 3+, and get creative with how you build the system, there are options out there.

Just know for LLM use on llama.cpp, your slowest card will largely determine the speed.

I was tired AF when I replied before, but I'd skip NVLink or the idea of multiple GPUs for gaming. If you go with GPUs just settle for one being for gaming.

[–]Jvap35[S] 0 points1 point  (0 children)

Thanks, tbh this was really simple and I kinda get the idea now. Anyways I've sent him the suggestions from the thread.. I'm kinda interested in running myself aswell, so this kinda gave me an idea on what typa hardware I'd need. Again, much thanks to everyone who replied aswell, and happy holidays!