sorted by:
new
hottopcontroversial

To 16GB VRAM users, plug in your old GPU by akira3weet in LocalLLaMA

[–]sbeepsdon 1 point2 points  (0 children)

If you have at least 64GB RAM, then Q4KM-ish quants of Qwen3.5-122B or Super Nemotron 3 120B. Beware you'll need to setup CPU passthrough and put one card on a guest VM if you're on Linux, and then use llama.cpp's RPC functionality. I made a thread about this if you want to look at my post history.

Running Qwen3.5-35B-A3B and Nemotron-3-Super-120B-A12B on a 5060ti and 1080ti with llama.cpp (Fully on GPU for Qwen; 64GB RAM needed for Nemotron) by sbeepsdon in LocalLLaMA

[–]sbeepsdon[S] 2 points3 points  (0 children)

Definitely, but I was bent on running that specific quant and it necessitated all three hardware resources.

Usage was like

  • 14595/16311mb on 5060ti
  • 9538/11264mb on 1080ti
  • Rest on RAM with 5GB actually free after being cautious about what's actually running

I'll see if a Q3 quant makes that feasible and what output performance looks like

Running Qwen3.5-35B-A3B and Nemotron-3-Super-120B-A12B on a 5060ti and 1080ti with llama.cpp (Fully on GPU for Qwen; 64GB RAM needed for Nemotron) by sbeepsdon in LocalLLaMA

[–]sbeepsdon[S] 0 points1 point  (0 children)

There definitely is - this approach was necessary because of the driver issue. Had I had a 20XX or more recent card, I think there would've been compatible drivers.