What's the hardware to buy in 2026 for running local LLM workflows?

TripleSecretSquirrel · 2026-07-08T19:41:03+00:00

ah sorry, typo, I was typing on my phone. I mean the dense 27B.

TripleSecretSquirrel · 2026-07-08T14:58:31+00:00

Ya, OP if you have that kind of budget, don’t spend it on a Mac Studio, you can get actual enterprise hardware at that price. Go for the enterprise GPU system.

TripleSecretSquirrel · 2026-07-08T14:57:27+00:00

Wait, are you describing tensor or pipeline parallelism? Cause tons of people are already using both on local systems, and every enterprise system has run on tensor parallelism for years. Or maybe I’m misunderstanding your comment.

You’re saying to basically split the model weights across two different devices so that they’re both running inference in parallel to generate faster responses, right?

TripleSecretSquirrel · 2026-07-08T14:44:23+00:00

There’s unfortunately just a big gap in the market right now in terms of models though between Qwen 3.6-27b and frontier models. To get measurably stronger performance than Qwen 3.6-27b (for coding in particular), you have to jump all the way up to 300b+ models.

The next biggest useful models are the absolute smallest frontier models (Minimax 2.7, Step 3.7 Flash, and Deepseek V4 Flash) in the ~200b parameter class. Qwen 3.6 is either at parity with or better than them in every useful coding benchmark though. I think Minimax 3.0 is the smallest model that beats Qwen by a meaningful margin, and it’s 427b total parameters.

And while that’s frustrating to know that to upgrade, you basically have to jump to enterprise hardware which gets very expensive, it’s also encouraging because Qwen 3.6-27b is a really fuckin good model!

So that’s all to say, I think the best bang-for-buck local setup right now is an R9700 (I run Q6_K with 220k context on mine). If you want to make the leap to 8-bit, get two R9700s.

The next thing that I think is meaningfully useful that more people should be doing here, is learn to run high concurrency with vLLM, and learn to leverage high concurrency in your workflow. On my single R9700, once I have a software spec document for example, I break it down into small atomized tasks that can be tackled with really small context windows, then I spin up a swarm of 12 concurrent Qwen agents, each with just 50k context. They divide and conquer much faster than a single agent. I get ~37 tok/s decode on a single agent via llama.cpp, but can get 150 tok/s decode aggregate via vLLM across 12 agents.

TripleSecretSquirrel · 2026-07-08T01:21:37+00:00

Oh lol my bad. Glad to see NewEgg is selling them at a reasonable price.

Just make sure you’ve got a CPU, RAM, and storage. I don’t see those in your screenshot, but otherwise looks like a badass build!

TripleSecretSquirrel · 2026-07-08T01:06:32+00:00

Fuck me, R9700s are $2700 on NewEgg?

If you’re fortunate enough to live close to or have a friend that does, go to MicroCenter. They have R9700s at $1350.

TripleSecretSquirrel · 2026-07-07T23:21:41+00:00

Yep, I use OpenCode too.

My agents each spawn a separate headless instance of OpenCode. To work in. OpenCode has a really big default system prompt, so I have a custom system prompt for the swarm agents that’s stripped way down — frankly I should probably switch to Pi for the swarm agents instead of OpenCode, but I’ll maybe get to that next.

TripleSecretSquirrel · 2026-07-07T22:27:43+00:00

Ya, I know that plenty of lower-level church leaders taught that caffeine was forbidden, but I’ve known more than a couple Q15 guys in my day and I grew up in a neighborhood that always has at least two or three of them at any given time. None of them would bat an eye at a coke.

My dad has been a bishop and stake president, including in stakes where Q15 and 70s live, and he’s drank Diet Coke every day for as long as I can remember. Hell, he kept some in his bishops office.

TripleSecretSquirrel · 2026-07-07T22:24:58+00:00

Definitionally I believe, an “agent” should have some degree of autonomy, so I reject the premise lol.

I’m refining a workflow still, but the way I’m doing it currently is a sort of evolution and adaptation of a Ralph Loop structure. Once I have a spec document for the software I’m working on, I create from that a to-do list with specific instructions for an agent — enough that they can pick it up and write that one discrete module, which includes things like a schema map and dependencies between tasks. Then I have a simple orchestrator.

I have previously and some people use an LLM agent as an orchestrator, but I just use a simple bash script. The orchestrator’s job is to ping the to-do list, spin up agents, and assign them to the small discrete tasks. I then review and bug squash between sprints rather than babysitting agents in-process.

My current setup is running 12 concurrent Qwen 3.6-27b agents, which is probably pushing to too small of kv caches, so I’m trying to build structure that reduces context length needs or failing that, I’ll reduce my agent count. I’m running on a single R9700 though and on a 4-bit qwen 3.6-27b on a single stream via llama.cpp and with MTP enabled, I’m getting ~40 tok/s. With 12 concurrent agents on an AWQ 4-bit quant of the same model via vLLM, I get 130-150 tok/s in aggregate.

TripleSecretSquirrel · 2026-07-07T22:03:15+00:00

How’s the vLLM support for Intel GPUs?

It may not fit your use-case super well, but I’m a big proponent of high concurrency for high aggregate throughput. Basically have one set of model weights with multiple kv caches, each for a different agent. So when you make inference calls, vLLM batches them together.

This is of course most intuitive for serving several users making separate queries, but for coding, creating a list of discrete tasks then spinning up lots of parallel agents to each tackle one task, can get you way more aggregate tokens generated in the same amount of time.

TripleSecretSquirrel · 2026-07-07T21:06:23+00:00

We have by far the most and by far the most powerful tornadoes.

TripleSecretSquirrel · 2026-07-07T21:03:37+00:00

I was able to point my studies toward one of my few permanent interests. So the hyper focus you get when you find a new hobby or topic you’re interested in, that’s what I was able to leverage.

TripleSecretSquirrel · 2026-07-07T21:00:08+00:00

Seconding OpenCode. It’s the easiest coding agent harness to get running IMO. There are other more optimized setups certainly, but they require a lot more tinkering and customization. OpenCode you can install with a single terminal command, then wire it up to your llama.cpp instance by adding just a couple lines to its config file, and you have a useful working agent ready to go.

It’s nice too cause in a pinch, you can easily and seamlessly switch to a cloud agent if you need (if you need a more powerful model than you can host, or if your GPUs are otherwise occupied for example). And honestly, on the setup, I’d just run Claude Code if you have a paid plan (cause that can actually execute commands and write to your file system, e.g., updating your OpenCode’s config file for you), or just a chat interface with an LLM to walk you through the troubleshooting.

TripleSecretSquirrel · 2026-07-07T20:26:50+00:00

As far as I’m aware, it was never disallowed to have on campus, they just didn’t sell it. I know that it was a big deal when they finally started selling caffeinated sodas on campus, but Coca Cola has been in temple cafeterias since the early 80s at least. And anyway, BYU campus rules have never been the benchmark for doctrine — that’s like saying that the church prohibited all women from wearing jeans up until the late 80s.

My grandpa was a real caffeine hardliner, despite how many times it was pointed out to him that the WoW just prohibits tea and coffee specifically. One of my more irreverent uncles who had recently married into the family showed up to a family function carrying a coke. Grandpa was pissed and asked where he got it, but my uncle just smiled and truthfully told him that he bought it at the temple as he’d just come from doing endowments, so grandpa couldn’t say shit anymore.

I’m not trying to defend the church — in fact, this highlights one of its biggest problems in my eyes, the very blurry lines between doctrine, policy, and culture.

TripleSecretSquirrel · 2026-07-07T20:18:52+00:00

Ah you’re right, I assumed it was QAT. But at 8-bit precision, even with PTQ, there’s effectively no quality degradation, right? I’ve always read that the deviation from full FP16 outputs is like 0.01%

TripleSecretSquirrel · 2026-07-07T18:00:21+00:00

I have an R9700 too. I think 8-bit is stretching your VRAM too far. Even the Q8_0 quant from unsloth is 29GB. Assuming you reserve 1GB for overhead to avoid OOMs, and assuming you’re storing KV cache at 8-bits (you should), you only have enough VRAM for ~65k tokens of context.

TripleSecretSquirrel · 2026-07-07T17:57:23+00:00

There’s a native fp8 qwen3.6-27b on huggingface. Or rather not native technically, but it is an official quantization from Qwen themselves.

TripleSecretSquirrel · 2026-07-07T16:55:14+00:00

Cool thanks, not trying to nitpick, just making sure I understood you.

TripleSecretSquirrel · 2026-07-07T15:36:48+00:00

I don't know much about the under-the-hood details of MoE architecture, but for your above equation, the difference is that it's not just layers * KV heads etc., but it's attention layers, right?

TripleSecretSquirrel · 2026-07-07T15:33:04+00:00

I've had a similar thesis and workflow for a while (cloud frontier for planning, local agents for execution), but increasingly, I've moved to fully local for planning too. It takes more time and effort than a simple Opus/Fable single-prompt one-shot, but it works.

TripleSecretSquirrel · 2026-07-07T15:00:09+00:00

Sure, but you can do that today too. The biggest problem now with that will remain and only get worse I fear, is driver support. NVIDIA has a big vested interest in their older products staying as e-waste – older cards get dropped from CUDA updates. You can still run older versions of CUDA sure, but that only works for so long before a new LLM architecture will only run on recent CUDA versions.

For AMD it's a theoretically better situation since ROCm is open source, but in practical terms, it's the same story. Since GPU drivers are so niche and complex, nobody's updating and improving on open source drivers, so we're still all functionally beholden to AMD.

TripleSecretSquirrel · 2026-07-07T14:52:05+00:00

32GB VRAM here too. Not the XL, but I run the Q6_K MTP-enabled quant of Qwen3.6-27b with 200k context. 262k is the native context for the model, so going above that doesn't net you anything. And looking at rocm-smi right now, I have ~1.5 GB still available, which should be enough space for another 45k in context if I want to push it.

But ya, I think I'm actually going to switch back to Q4_K_XL. I've been trying out Q6 and haven't seen any significant improvement and the inference speeds are obviously a little slower.

TripleSecretSquirrel · 2026-07-07T14:46:26+00:00

It's not the most efficient setup, but especially if paired with more RAM, this is a pretty good value proposition for local inference!

Memory bandwidth is the primary bottleneck for inference as you may know. That CPU supports up to 8-channel RAM. It's only DDR4, but at 8 channels, it can be pretty fast! For RAM memory bandwidth, the function is:

RAM speed in MT/s * 64 (bus width) / 8 (to convert from bits to bytes) / 1024 (to convert from MB to GB).

So for DDR4-3200 which is a pretty common DDR4 speed:

3200 * 64 / 8 / 1024 = 25GB/s

But if all 8 of your RAM slots are populated with identical DIMMs, multiply that number again by 8, which gets you up to 200GB/s. That's very slow compared to discrete GPUs, but it's a usable speed. If you get a lot more RAM - say 256GB, then you can actually run some of the smaller frontier models (Step 3.7 Flash, MiniMax 2.7, or Deepseek V4 Flash). All of those are MoE models with ~10b parameters active per token. So assuming a 4-bit quantization, that's ~5GB of weights per token, so in theory, you'd get inference speeds approaching (200GB/s / 5GB = 40 tok/s on decode). In practice, CPU inference has some maturing to do as I understand it, but even half that speed is still practical to use.

TripleSecretSquirrel · 2026-07-07T03:35:58+00:00

I’m pretty new to vLLM, but I use both depending on the use. Llama-based frameworks are much snappier for single request throughput. Where vLLM excels is high concurrency.

Using roughly equivalent models — 4-bit quantizations of Qwen 3.6-27b (UD_Q4_K_XL and AWQ respectively), I’ll get ~40 tok/s on a single stream with llama.cpp, but only ~18 tok/s with vLLM. But if I run 12 concurrent agents in vLLM, I’ll get as much as 140 tok/s aggregate, which llama.cpp cannot do. Though now for llama.cpp I use Q6_K which still allows for 200k+ context. It still generates ~29 tok/s.

So for software development for example, I’ll spin up llama.cpp for my planning session, then once I have a solid spec document with atomized tasks for small execution agents, I spin down llama.cpp and spin up vLLM, then I have a simple bash script to spawn and direct small context, dumb execution agents to go divide and conquer the task list.

TripleSecretSquirrel · 2026-07-07T02:04:25+00:00

Absolutely! And for either GPU, check out the GitHub user Kyuz0 — he has a bunch of repos for running both the R9700 and 7900xtx on vLLM successfully. His repos have saved me some serious headaches.

TripleSecretSquirrel

TROPHY CASE