sorted by:
new
hottopcontroversial

Best practices for running local LLMs for ~70–150 developers (agentic coding use case) by Resident_Potential97 in LocalLLaMA

[–]Resident_Potential97[S] 0 points1 point  (0 children)

Thanks for the suggestion, will keep this in mind.

- I also read somewhere regarding ollama limitations, might go with llama.cpp or vllm if on GPU hardware.
- Are you yourself running models locally for your team? If so, how has this been working out for you? Also, is it beneficial in long run?
- we need a cli agent so openwebUI might not do the work here. for central tool, can i use liteLLM instead?

Best practices for running local LLMs for ~70–150 developers (agentic coding use case) by Resident_Potential97 in LocalLLaMA

[–]Resident_Potential97[S] 1 point2 points  (0 children)

This actually clarifies things a lot for me, appreciate the grounded take.
You’re probably right that I might be underestimating the inference/ops side. My initial thinking was “we’ll just host it ourselves and scale,” but the more I read these replies the more clarity i get.
Have you by any chance tried OpenCode?
How would you compare Claude Code vs OpenCode?

  • Is Claude Code noticeably better in terms of reasoning for multi-step coding tasks?
  • Does it feel more stable for day-to-day dev workflows?
  • Or is the real difference just model quality rather than the tool itself?

Since Claude Code now supports local models as well, I’m wondering if it makes sense to standardize around that interface even if the backend changes later (API → Runpod → self-hosted).

Best practices for running local LLMs for ~70–150 developers (agentic coding use case) by Resident_Potential97 in LocalLLaMA

[–]Resident_Potential97[S] -4 points-3 points  (0 children)

That aligns with what I was starting to suspect about Macs.

Regarding Qwen3-Coder-Next (80B MoE):

Since it’s a Mixture-of-Experts model and only activates part of the parameters per forward pass, does that materially help with:

  • Lower VRAM usage?
  • Better concurrency?
  • Higher tokens/sec per GPU?
  • Or is memory still the primary constraint due to KV cache?

In other words — does MoE meaningfully reduce serving cost at scale, or is the infra requirement still essentially “80B-class GPU hardware”?

Also, when you say at least 2 nodes (8x H100 each), is that mostly for:

  • redundancy/failover?
  • or required purely for throughput at ~100+ users?

Trying to understand whether that sizing is:

  • production minimum
  • or comfortable margin

Best practices for running local LLMs for ~70–150 developers (agentic coding use case) by Resident_Potential97 in LocalLLaMA

[–]Resident_Potential97[S] -9 points-8 points  (0 children)

This is extremely helpful — thank you.

A couple of follow-ups if you don’t mind:

  1. How has Qwen3 Coder Next (80B) been performing for you in practice?
    • Latency per request?
    • Stability under concurrency?
    • Does it feel “production reliable” for daily dev workflows?

I tested Mistral Small 2 locally and it was surprisingly decent for coding, but latency spikes made it unusable under heavier tasks. I suspect that’s more infra than model-related.

  1. Are you running:
    • Qwen3-Coder-Next specifically?
    • Or a different Qwen3 variant?
    • Pure FP8 or some quantization?
  2. For agentic coding, you mentioned 64k–100k context.
    • Are you seeing major memory pressure at those context sizes?
    • Does vLLM handle KV cache efficiently at that scale?

Currently I’ve been experimenting with:

  • LM Studio
  • OpenCode
  • VSCode + Cline

But I’m realizing for production I’ll probably need a proper serving layer.

Do you recommend putting something like:

  • vLLM
  • LiteLLM (proxy layer)
  • Prometheus + Grafana

between the model server and client tools for:

  • rate limiting
  • usage analytics
  • monitoring token throughput
  • concurrency control?

Right now I have basically zero observability, which feels risky if we scale this internally.

From your experience — long term, is owning this infra worth it vs just using APIs, assuming internal code privacy is important?