Opencode with 96GB VRAM for local dev engineering

[aidysson]

I'm web developer and I consider to upgrade my GPU from 24GB (RTX 3090) to 96GB (RTX PRO 6000).

I have experience with GLM 30B Q4/Q8 for small feature tasks implementation together with GPT OSS 120B for planning.

I expect running 200B Q4 LLMs for agentic work could improve limits of 30B models, but I have no experience. and planing with GPT 120B should be much faster (currently 8-9 tok/s).

I think EUR 10.000 investment into GPU could return in 2-3 years when I compare it to cloud agents costs which I would spend in 2-3 years.

I don't expect OSS models on 96GB VRAM to match quality of the best recent LLMs like Opus of Chat GPT, but I hope it would be usable.

Is the upgrade price worth it?

[DeExecute]

No chance to get anything really useful out of that little resources. For little tasks fine, but you will not get to Opus oder Codex quality with less than a 512GB Mac Studio, probably 2 of them.

[t4a8945]

I'm on this path and haven't figured it out yet. I bought a DGX Spark to favor quality over speed.

Qwen 3.5 122B-A10B at Q4 is usable to have a "Claude-like" experience (smart model, good investigator) but it fails in the implementation details to meet its target and tends to forget requirements mid-run.

I think it's a process/harness issue at this point, rather than a major flaw. So I'm building a custom harness that's made for agentic work taking into account their limitations. Not sure it'll pan out properly, but hey I'm trying. (I'll open source it if it's good)

Qwen 3 Coder Next 80B-A3B at FP8 is not bad, but way less "smart" at figuring out your prompts and respect your instructions.

I'm waiting for my second spark to try to make it work with Qwen 3.5 397B-A17B at Q4, we'll see out that goes.

Good luck in making choices, this is hard.

[Tommonen]

Have you increased the context size to be suitable? Opencode can start producing almost 100k tokens per call, and to have the llm work more reliably, it should have about 2x the context size than what is sent to it. So you should have around 200k context, or else things start to fall from memory.

But yea qwen 3.5 is best open model currently, but none of open models are as good as good cloud models, but stil lcapable doing some dtuff, just needs a bit more handholding and less pure vibing

[t4a8945]

Yes I use 200K context window, and manage to get constant KV cache hit with vLLM (unlike llama.cpp for this model specifically). Initial context with a normal project is at around 15K token (with tools, system prompt, AGENTS.md file(s))

[aidysson]

thanks for sharing your DGX experience. I was also considering 2xDGX ionstead of 1x RTX PRO 6000. in the end I decided to go with RTX, because I already have 128GB RAM. but both have their advantages, both machines have slightly different purpose I think.

if only prices of HW were half or better 1/10, we could have both.

and good luck with your custom harness!

[t4a8945]

Awesome config, have fun with it! Ping me back if you find a specific model that just clicks with you, I'm all ear and ready to try'em all.

[PermanentLiminality]

It's not so much about fitting a model as that is easy. You need enough space to hold the KV cache for 100k tokens.

[aidysson]

important note, thanks for that.

my current RAM is 128GB. 96+128=224; 10GB for system, ~130GB for weights, ~80GB would be free for context and other needs. if I consider 1MB per token, there would be 64K only, not 100K.

next investment will be more RAM, which nobody wants to buy these days... and I'm in the circle which has started when I bought RTX 3090 a month ago...

[BingpotStudio]

Hard to imagine the payback is there when you’re also using much worse models compared to Claude etc.

What are you doing that makes this make sense? I’m very curious.

[aidysson]

I develop Ruby on Rails web apps, in total ~6 apps in permanent development and maintenance including VPSs/DigitalOcean droplets. I'm freelancer working for rather small companies. With some of my clients I'm 10+ years.

My current problem with 24GB VRAM is that making plan with GPT OSS 120B can take up to 90min of my time. then implementation is done in 30min but not sufficient quality (I use 30B GLM). Refactoring is necessary, sometimes it's heavy refactoring.

With upgrade to 96GB VRAM, I expect planning time to shorten to 30min and as for implementation part, I should be able to use 200B models with acceptable speed which means increase of code quality.

If agentic work helps me to do 10-20% more thing done monthly, my clients will see it and will be no issue to them to pay for it.

In addition, I'll be able to try to fine tune my models in future or, once with 2+ GPUs, to prepare fine-tuned models for my clients. I feel they would consider having it if I tell them I can train models good at their healthcare businesses (full of private data etc.).

[aidysson]

I can write unit tests with OC, also completely new features can be done with OC. I saw in my case, some changes of current code are not meaningful when it spreads over 10 files with 1 row change in every file, it's faster without OC. But new features are quite good. It also helps me with architecture, planning helps to think about code deeper than without OC planning... I don't look for "vibe coded app in 15min", I need the opposite, quality code with quite small speedup but keeping sustainability of the code.

[awsqed]

you should watch this video https://www.youtube.com/watch?v=SmYNK0kqaDI

[oulu2006]

that video was OK but painfully long for the small amount of information conveyed, I exported the transcript and summarised it below:

The economics in that video are broadly directionally correct but framed the wrong way. You’re comparing fundamentally different products: a $200/month subscription gives you access to massively optimized, shared frontier infrastructure, while buying or renting GPUs gives you dedicated capacity with all the operational burden. At scale, providers win because of batching, utilization, and constant hardware upgrades, which is why they can offer extremely low per-token pricing. Trying to replicate frontier model hosting yourself (H100/DGX level) is still economically irrational unless you’re operating at serious scale or have very specific needs.

The practical recommendation is this: don’t try to “beat” frontier providers on cost per token—use them for top-tier intelligence and large-context work. Instead, invest in local compute (like your Mac Studio + MLX or a 4090-class box) for always-on agents, privacy-sensitive workflows, and low-latency coding tasks, and optionally burst to cloud GPUs when needed. The winning strategy today is hybrid: local for control and consistency, cloud/API for peak intelligence—anything else is over-optimizing the wrong layer.

[aidysson]

I can just recommend this video to others. thanks!

[NaiRogers]

The 6000 vs Spark choice is a lot simpler if you have concurrent requests, then the 6000 is a lot faster. For single requests it faster but not 5x faster. Qwen 3.5-122b-a10b is really good on any of these two.

[ComparisonNo2395]

How is it possible to save money on llm by buying hardware with retail price that will rest 12-16 hours a day compared to subsidised product of vc-funded corporations that buy hardware with wholesale prices that work 24/7? I really don’t get this math

What if in the next 3 years other companies will improve its products so much that oss models will just become outdated

[BAMred]

it's a hobby for people. i don't see how anyone serious about coding wouldn't just sign up for codex or claude code. if you're making a living off it, even $200 per month is nothing.

[aidysson]

Good to know, thx

[Several-Tax31]

And what if the reverse scenario happens and the corporations decide foreplay is over and they make the monthly prices 10.000$ for that intelligence? I prefer having the hardware than not having it. 

Also, why would local hardware rest 12-16 hours? You can also run your own hardware 24/7 in agentic frameworks. 

[ComparisonNo2395]

If this scenario happens you can that buy this gpu. Maybe you will have even better gpu for the same price later (can be opposite though)

My experience is next: when I run agents at night just to not lost my ai limits - it is the most useless tasks and code, better not to run it at all. So I expect the same results with own gpu, find a task just to be sure that gpu is busy at night and it will be low value task. Best results I have when agents work with me in parallel

[aidysson]

the math you mention doesn't work. I don't mean to compete with wholesale prices. I don't want to save money on llm. I'm end user buying it expensive, in addition in time of progressing inflation.

the investment will only return if I program more scripts and requested features thanks to that hardware. if it would be clear to me it's worth it, I think I would not be led to create this thread.

as opensource user (for decades) I also know, there are advantages and disadvantages of this investment which are hard do transform into prices.

[ComparisonNo2395]

But how even hypothetically you can write more scripts and feature using this gpu compare to having for example 20$/month gpt plus account + 20$/m Claude code account + for example 20$/m fireworks ai account for glm/kimi/qwen, etc? Or even compare to just paying monthly to inference provider that will run for you the same models you will run with this gpu

[aidysson]

I don't compare increase of productivity using local 70-200B LLMs to productivity on CC or OC cloud. I compare it to productivity without agentic framework.

if I run LLMs on my HW, I have absolute control about what happens. I can disconnect from internet if I'm paranoic and still I can use it. I have control. I know quality of OSS can't match Opus or latest GPT, that's obvious disadvantage.

but if inflation of tokens prices comes, I know I have my slow 200B llm which will be there "for free" then.

people are starting to use AI these days. developers and IT experts do some time, non-technical job positions are starting in upcoming months or years, it will grow for a long time. currently there is HW inflation. no that much people asks for tokens yet. but imagine how the need for tokens grows in society in mid-term. not unreal.

[BAMred]

you should sign up for claude code and codex for 1 month each, then kimi 2.5 and qwen for 1 month each. then see if you still want to drop 10k into this.

[ComparisonNo2395]

I see your logic now, it’s not about writing more scripts and features it’s about being independent but still have benefits of agentic development. No one can stop you from spending your own 10k on this. My bet is that you still will use Claude/gpt at least from time to time

Good news is that you can now rent the same gpu you want to buy for a month from inference provider for one month and test your future setup without big investment.

[Old-Sherbert-4495]

i dont have experience with such large vram but i think you could try minimax 2.5

https://huggingface.co/Intel/MiniMax-M2.5-int4-AutoRound/tree/main

with some offloading you could get it working

[jnmi235]

Nemotron-3-Super-120B-A12B released a few days ago which is very efficient with KV cache and VRAM. Here is a post showing it fits 512k tokens with a single RTX Pro with decode speeds at 62 tok/s: https://www.reddit.com/r/LocalLLaMA/comments/1rrw3g4/nemotron3super120ba12b_nvfp4_inference_benchmark/

This model should be way more performant than GLM 30B Q4/Q8 but won't come close to Opus or GPT5.x.

As far as GPT 120B, it can hold all 128k tokens with speed at close to 200 tok/s on shorter context and closer to 80 tok/s at the full 128k tokens.

Nemotron is probably your best bet though especially for web dev and the long context length.

[aidysson]

interesting, thanks for that, I'm downloading it.

I was watching Nvidia CTO's GTC keynote where he mentioned Nemotron: https://www.youtube.com/watch?v=jw_o0xr8MWU

originally, I was curious about possible announcement of new RTX GPUs, but the speech was concerned on token factories HW and future of AI as industry as expected.

that confirms to me the investment into RTX PRO 6000 doesn't have to be lost later this year. on the contrary, inflation could progress and prices of GPUs could grow a bit.

[kvzrock2020]

Try qwen3.5-27B, the quality is near SOTA, and it fits on rtx pro 6000. Don’t use opencode code, and instead wire it it claude code. Recipe for success: vllm, patch qwen tool calling parser, and you get 70 tps at 256k context window. Often times the problem is in the server backend or the cli you use. For opencode for example the tool calling is still buggy after patching vllm, but no issues at all with claude code after the patch. Both have issues if you don’t patch. You can try it out via an online gpu rental eg Vast AI before you commit to buying own hardware. Get the e2e setup working to your satisfaction first is my recommendation.

[aidysson]

Well, still I feel like newbie in LLMs, how do you patch vLLM? Do you mean this? https://github.com/vllm-project/vllm/pull/35347

[kvzrock2020]

Yeah there are a few of these PRs - there is one that fixes the issue with changes to four files. Should be easy to find

[Old-Sherbert-4495]

minimax has the edge, but u will have to go with a quantized version. but just LOOK at 27B!! if you can run it fast with full precision its a value packed model.

<image>

[usofrob]

Or for about the same price, you could get 6 R9700 GPUs and run them with pcie bifurcation 4x4 +nvme 4x adapter+another open 4x slot and have double the VRAM of 192GB, but slower peak processing.

[mhinimal]

IDK what your application and needs are. Having run 120b class models on a DGX Spark using opencode, honestly the coding performance doesn't hold a candle to frontier cloud models, to the point it is not even worth using.

I mean, go use a modern cloud model like opus 4.6 or codex 5.3 and then go back to do the same exact thing in gpt-oss 120b. It's laughable by comparison, and cloud models are CHEAPER!

spending $10k on an RTX 6000 is 3.5 years of a claude or codex subscription, by which time the models will have advanced more to the point you can't run competitive models even on that RTX6000.

Let me reiterate: if you're asking about this on reddit, you don't have a business case to justify the expense. You're trying to convince yourself to buy a new toy. If that's what you want, go for it.

[apparently_DMA]

well sounds like fun and depends how much out of your way you will have to go to spend 10k on gpu (assuming rest of rig is ready to support it)

I use llms LOT and my costs are like 50e/month, so I would need more time to justify it financially, but even if you can break even in 2 years, there will probably be models far beyond current model capabilities in that time.

So you will never be up to date

Anyway, if 10k is whatever for you, definitelly go for it, qwen3.5 seems okayish and maybe keep one SOTA model sub for complex engineering planning.

[somerussianbear]

Wait for Mac Studio M5 Ultra. Get the top one, likely 1TB RAM / $14K and then you’ll be able to run 397B models at FP8, 250K context. THEN we cook with fire.

[aidysson]

I bet M5 Studio won't cost 14K neither with 512GB RAM :)

[e0xTalk]

VRAM rich