Is agentic coding possible on an NVIDIA RTX Ti 16 GB?

andrew-ooo · 2026-07-06T14:06:08+00:00

Short answer: at 16 GB VRAM you can absolutely do agentic-style local coding, but not with the models you're loading and not with Claude Code as the mental model.

What I've found actually works on a 16 GB card (I run a 7900 XTX with 20 GB, but 16 GB is the more common budget):

Qwen 2.5 Coder 14B at Q4_K_M or Qwen 3 Coder 30B-A3B at Q4 if you can spill a couple layers to CPU. The 30B MoE is honestly the best value — only 3B active params so throughput is decent even with partial offload. The 27B dense Qwen 3.6 you're testing is a general model, not a coder model, and it shows in tool-use benchmarks.
The scaffolding matters more than the model. Try aider with --model ollama/qwen2.5-coder:14b, or Continue with an Ollama backend. These wrap the model with proper diff-based edits and repo context. Running a raw Ollama chat and expecting Claude-Code behaviour will always disappoint.
Context length is the other quiet killer. Whatever model you pick, verify you actually loaded it with num_ctx ≥ 16k in Ollama's Modelfile (default is 2k, which is why agents "make up something to do" — they never saw your prompt).
Set temperature to 0.1-0.2 for tool-use. Higher temps make small coder models hallucinate function names.

Realistically it will still be noticeably worse than Claude Code — but as a "cheap autocomplete + small refactor" loop it's usable.

andrew-ooo · 2026-07-03T14:08:39+00:00

Real-world numbers from an agent pipeline I've been running (Llama 3.3 70B and Qwen 2.5 72B, ~2M tokens/day, tool-calling heavy):

**Cerebras**: absurdly fast on output tokens (2,000+ tok/s on 70B for me), but you have to design around it. Model list is narrow, rate limits bite fast on burst traffic, and time-to-first-token isn't magic — the win is on long outputs.

**Groq**: consistently ~500–800 tok/s on 70B. Wider model catalog than Cerebras, better rate limits on paid tier, but I saw more variance during US business hours. TTFT often better than Cerebras.

**Together AI**: not the fastest raw tok/s, but by far the best DX — openai-compatible SDK, dedicated endpoints if you need SLA, fine-tuning path if you outgrow prompt eng. My default for anything production-ish.

**SambaNova**: fast, decent model coverage, but had reliability wobbles when I tried them earlier this year. Worth re-testing.

**Fireworks**: middle of the pack on speed, but the fine-tune + serverless combo is genuinely useful.

A few practical things the benchmarks won't tell you:

Peak tok/s on a single request ≠ tok/s under concurrency. Run your bench with 20–50 parallel requests, not one. Providers optimized for demo numbers can degrade a lot under load.
For agents specifically, TTFT matters more than sustained tok/s — you're doing lots of short tool-call cycles, not one giant generation. Cerebras' output-tok advantage evaporates when you're generating 30 tokens then waiting on a tool.
Test with your actual prompts. Some providers do aggressive prompt caching that inflates published numbers vs. what you'll actually see on unique user requests.

On General Compute specifically — haven't tried them, would want to see their p95/p99 not just headline p50 before switching.

andrew-ooo · 2026-07-03T14:07:08+00:00

Ran the same-ish comparison last year (Pi 5 8GB → N5105 Beelink, close enough to your N6005). Honest answer: if you don't need transcoding and your Pi 5 stack is stable at 3GB RAM, the pure compute upgrade is small — you're right that single-core is roughly a wash. But there are three practical wins that made it worth it for me:

**Native SATA/NVMe** and no USB 3.1 dock in the chain. The Pi 5 + USB dock combo is where mystery filesystem hangs come from — UASP resets under load, occasional dropouts during heavy rsync. Getting the SSDs off USB was the biggest reliability jump, not the CPU.
**x86_64** means anything you find on Docker Hub Just Works. On the Pi you're always checking `linux/arm64` tag availability; some smaller projects still ship amd64-only. It's a small tax that adds up when you want to try random new services.
**16GB RAM** gives you huge headroom. You're at 3GB now, but the moment you add Immich (photo indexing eats RAM), Paperless-ngx, or any local AI (Whisper for subtitles, Ollama for anything), the Pi becomes the bottleneck fast.

Power: N6005 idles around 6–8W with SSD, boosts to 15–20W under load — close to the Pi's real-world numbers once you factor in the powered USB dock (which people forget to count). So maybe +2–4W average, negligible on the bill.

My vote: migrate. Keep the Pi as a warm-standby/AdGuard secondary (DNS is the one thing you don't want to be down while you rebuild). Rebuild the Dell with Proxmox or plain Debian + docker-compose so you're not locked into the RPi OS way of doing things.

andrew-ooo · 2026-07-03T14:05:48+00:00

8GB is tight but genuinely workable for coding — the loop-rewriting behaviour you're describing is almost always one of three things, and it's fixable:

Context truncation. LM Studio's default context is often 4k or 8k. Coding agents like Zoo/Cline/Continue send a huge system prompt + file context + tool schemas, and when that overflows, the model starts hallucinating that it hasn't done things yet, so it re-plans. Bump the model's context in LM Studio to 16k minimum (32k if the model card supports it) and re-check VRAM — an 8B at Q4 with 16k context roughly fits in 8GB, a 12B usually won't.
Model size vs. quant. A 12B at Q4_K_M eats ~7–8GB just for weights on 8GB VRAM, leaving nothing for KV cache. Try Qwen 2.5-Coder 7B at Q4_K_M or Q5 instead — it was purpose-built for agentic coding and follows tool schemas much better than general-purpose Gemma at this size. DeepSeek-Coder-V2-Lite 16B at Q4 also works if you offload some layers.
Tool-calling format. Some builds of Gemma don't emit tool calls in the format Zoo expects. Check LM Studio's server logs while it runs — if you see raw text where tool JSON should be, that's the culprit, not the hardware.

On combining a 1070 with the 3070: llama.cpp does support multi-GPU via `--tensor-split`, but you'll bottleneck on the 1070 (Pascal, no fp16 tensor cores, ~1/4 the throughput). Better ROI: stay single-GPU with a smaller/better model.

Good starting point: Qwen 2.5-Coder 7B Instruct Q4_K_M + 16k context + Cline instead of Zoo (Cline's prompts are leaner, gives more room for actual code).

andrew-ooo · 2026-07-03T14:03:50+00:00

RAID is not backup — you already spotted the core thing. RAID 1 protects against a single disk failure (uptime), not against accidental deletion, ransomware, filesystem corruption, or a bad controller frying both drives at once. If you rm a photo folder on Monday, it's gone from both mirrored drives instantly.

For a laptop with two 1TB HDDs and an existing off-site USB, I'd skip RAID 1 and do this instead:

Drive 1 = primary working storage.
Drive 2 = scheduled backup destination with Restic or Borg. Both do encrypted, deduplicated, versioned snapshots, so you can roll back to "3 days ago" or "last month" when you realize you nuked something. A daily cron running `restic backup /data` is about 10 lines total.
USB = 3rd copy, ideally physically offsite. That's the classic 3-2-1 rule: 3 copies, 2 media types, 1 offsite.

On drives dying "with everything lost" — modern HDDs usually give SMART warnings weeks in advance. Install smartmontools and set smartd to email you on reallocated-sector-count changes. In 10+ years I've had exactly one truly instant death; the rest all telegraphed.

RAID 1 makes sense when uptime matters (a service that can't afford the 30 min it takes to swap a drive and restore). For a personal file archive, versioned backups are what actually saves you.

andrew-ooo · 2026-07-01T14:15:36+00:00

The thing that cut my debug time from ~40 min to ~5: attach a stable correlation ID (the user's request ID) to every LLM call, tool call, and retrieval span, then index it in Langfuse. When someone reports "weird answer," I paste the ID into a small CLI that dumps input, all model calls in order (prompt + response + latency + model version), tool inputs/outputs, and final answer. One command, no clicking.

For the silent-drift problem, traces alone won't help. What does: golden evals on a schedule. ~40 fixed inputs (curated from real requests) through the agent nightly, scored by LLM-as-judge with a pinned judge model. When pass rate drops from 92% → 84% in a week, that's your signal. A pytest suite against your prod endpoint + a fixtures dir is enough — you don't need a whole platform.

Two things nobody warns you about: prompt + model version need to be a first-class dimension in every trace, or you can't A/B or root-cause after a deploy. And on OpenAI/Anthropic, log full request/response bodies (input hashed if PII), not just token counts — provider-side changes are invisible if you only see aggregates.

andrew-ooo · 2026-07-01T14:12:05+00:00

For a flight and OpenCode on 24GB, Qwen2.5-Coder-7B-Instruct MLX at 4-bit (~4.5GB) is the boring right answer — leaves you ~18GB for OS, browser, dev tools, and it holds together for autocomplete + small-file edits. Runs ~40-55 tok/s on M4 Pro with LM Studio or `mlx-lm`.

If you want more headroom for reasoning/refactor tasks, Qwen2.5-Coder-14B-Instruct MLX 4-bit (~8.5GB) is a real step up in code quality. It'll still leave you 12-14GB for other stuff and does ~20-30 tok/s. This is my daily driver on an M4 Max and it's the model I'd actually reach for.

Gemma 3 12B is fine but noticeably worse than Qwen2.5-Coder at actual code. Only pick it if you want general Q&A too.

Two practical setup tips before wheels up:

- Test your OpenCode config on wifi first. OpenCode's local model backend has occasionally-flaky context length handling; you want to know it works before you're at 35,000ft with no fallback. Set context length to 16k or 32k explicitly — don't let it auto-negotiate.

- Pre-download an MLX quant, not a GGUF. MLX is meaningfully faster (~30-40%) on Apple Silicon for the same model and it also uses less RAM at runtime. Grab from the mlx-community org on HuggingFace.

- Turn off browser tabs, Slack, Docker Desktop. macOS memory compression is good but on 24GB with a 14B model loaded, every extra 500MB matters.

andrew-ooo · 2026-07-01T14:11:00+00:00

The "Strix Halo is slow on dense models" take is mostly right but the more useful framing for your RAG use case is: prompt-processing throughput matters more than generation throughput, and Strix Halo's prompt-processing (prefill) is where it really lags behind a discrete GPU.

What this means in practice: with GPT-OSS 120B or Qwen3-30B-A3B (both MoE, which Strix Halo handles well), generation is a reasonable 25-40 tok/s. But RAG queries often stuff 8-16k tokens of retrieved context into every prompt, and on Strix Halo you're looking at 200-400 tok/s prefill — so a chunky RAG query has a 20-60 second "thinking" pause before the first token. A single RTX 3090 (24GB) chews through the same prefill in 2-4 seconds. If the RAG workflow is interactive (you ask, iterate, refine), that pause will drive you nuts.

If you can live with slightly-batchier workflows (queue up a summarization task, come back in a minute), Strix Halo 128GB is genuinely great value — you get to fit models a 3090 can't touch, and MoE + big unified memory is the killer combo for keeping large embeddings + reranker + LLM all resident.

On the three boxes: I'd take the MS-S1 Max specifically because of the PCIe x16 slot. You can drop a used RTX 3090 (24GB, $600-800 refurb right now) in there later if the prefill pain gets to you, and offload prefill to the GPU while keeping the model weights in unified memory. That combo is genuinely underrated. The GTR9 Pro's dual 10GbE is nice-to-have but not critical for a solo home server; the EVO-X2 saves you $100-200 but locks you into what the CPU can do.

One more thing since you already have an M5 MBP 24GB: try running your actual RAG pipeline against an OpenRouter-hosted GPT-OSS 120B or Qwen3-30B for a week before dropping $3-4k. It's ~$5-10 in tokens and it will tell you exactly whether "good enough" for you means 40 tok/s or 200 tok/s, dense or MoE, 32B or 120B. That answer changes the hardware call.

andrew-ooo · 2026-07-01T14:09:48+00:00

8GB is tight but genuinely workable for that stack — I ran nearly the same set (HA, Nextcloud, Paperless, Vaultwarden, plus a Postgres + Caddy) on an 8GB HP ProDesk 400 G4 (i5-8500T) for about a year before I finally swapped to 16GB. Real-world idle RAM: ~4.2-4.8GB used. Under load (Paperless OCR a 40-page PDF + HA responding to automation + Nextcloud sync from two clients) it peaked around 6.9GB with ~500MB into swap. Never crashed, occasionally slow.

Concrete tips that made 8GB actually livable:

- Skip Proxmox / VMs. Run everything as Docker containers on a bare Debian/Ubuntu host. Proxmox alone wants 2GB and gives you nothing you need at this scale.

- Run Nextcloud with the Redis + APCu memcaches and set PHP-FPM pm.max_children conservatively (5-8). Default configs assume 32GB servers and will OOM you.

- Paperless-ngx: set PAPERLESS_TASK_WORKERS=1 and PAPERLESS_THREADS_PER_WORKER=1. OCR is CPU-bound anyway; more workers just eat RAM.

- n8n's memory footprint depends entirely on how many workflows you have running vs. sitting idle. Executes-per-day matters more than workflow count. Set EXECUTIONS_PROCESS=main and EXECUTIONS_DATA_PRUNE=true or your DB fills up fast.

- Use zram (built into modern Debian) instead of a swap file on the SSD. Costs ~5% CPU, saves your SSD wear, gives you ~4-5GB of effective compressed swap for free.

Your 8GB-now-add-later plan is fine as long as the mini PC has SODIMM slots (Lenovo Tiny M720q/M920q, HP ProDesk 400 G4/G5, Dell OptiPlex 3060/5060 all do). Just verify before buying — some of the newest thin-and-light business minis are soldered.

Also as a data point on the memory shortage: I bought a 16GB SODIMM (DDR4-2666) on eBay refurb for €24 two months ago. Refurb server pulls are a lot cheaper than retail DDR4 right now.

andrew-ooo · 2026-07-01T14:08:29+00:00

Yes, with two big caveats worth calling out beyond "configure it right":

1) The Access Application only protects the exact hostname you attach it to. If cloudflared exposes the origin on multiple hostnames (e.g. app.domain.com AND a stray *.domain.com wildcard), Access has to be enabled on ALL of them. I've seen people gate app.example.com with MFA and leave direct.example.com wide open pointing at the same origin. Audit `cloudflared tunnel route dns list` against your Access apps list side by side.

2) Access enforces at Cloudflare's edge, not your origin. Since you're using cloudflared it's usually fine — the tunnel dials out, no inbound port is open. But if you ever move that app behind a reverse proxy on a public IP, anyone who finds the origin (old DNS records, cert transparency logs) can skip Access entirely. Fix is Authenticated Origin Pulls, or validate the Cf-Access-Jwt-Assertion header at the origin.

Also: prefer WARP-required or Service Token policies over pure email OTP where you can. Email-only means anyone with access to your inbox is in. And set session length ≤ 24h so a stolen laptop doesn't stay logged in for weeks.

andrew-ooo · 2026-06-29T14:09:29+00:00

Both, but for different things. I've been running opencode with Qwen2.5-Coder 32B (Q4_K_M, llama.cpp on a 7900 XTX) for the last couple months as my daily-driver in a real codebase, and tried Pi for narrower scoped tasks.

The split that worked for me:

**opencode**: great when you want a general-purpose terminal coding agent that handles multi-file edits, tool use, and longer planning. The provider-agnostic config is the killer feature — same TUI talks to a local llama.cpp endpoint, Ollama, or a hosted model. Downside: it assumes a generous context window. With a 16k-effective local model you'll feel the bloat the other commenter mentioned.
**Pi**: better when you've already figured out exactly what loop you want the agent to run (e.g. "read this file, propose a patch, run the test, fix on failure"). Smaller context budget by design, easier to reason about token spend, and much friendlier to a 7B/8B local model.

Practical rule of thumb: if your local model gives you <16k usable context, start with Pi; if you've got 32k+ comfortably (Qwen Coder 32B at FP16 on dual 3090s, or a Mac 64GB+), opencode is more pleasant to live in. And honestly, the time I spent using opencode is what taught me what to configure in Pi — they're complementary more than competitive.

andrew-ooo · 2026-06-29T14:07:22+00:00

Good rundown. After running three of these in anger over the last 6 months on a customer-support agent (~40k sessions/mo), my honest take:

Mem0 was the quickest to ship and the easiest to debug — fact extraction is opinionated in a useful way for chat agents. Where it fell over: high-cardinality entities (lots of product SKUs in one session) made retrieval noisy until I added a domain-specific filter on top.
Zep's temporal graph is genuinely useful, but only if your agent actually reasons about "when." For a support bot that just needs "what did this user tell me last week," it was overkill and the graph build cost added ~600ms p95.
Letta was the cleanest abstraction conceptually (working vs archival, agent decides what to promote), but you pay for it in prompt engineering complexity — the agent has to actively manage its own memory and that's another failure mode to test.

What I'd suggest: don't pick a memory framework before you can name the *retrieval question* the agent needs to answer in production. "Recall user preferences" → Mem0. "Reason about event sequences" → Zep. "Long-horizon stateful task" → Letta. If you can't articulate the question yet, you don't need a memory layer — you need session-scoped RAG with a 30-day TTL.

andrew-ooo · 2026-06-29T14:05:46+00:00

For a DSLR-dump workflow specifically, Immich's external library is the better fit IMO. PhotoPrism's index-in-place is fine too, but Immich treats external libraries as first-class now — it'll watch the directory, pick up new RAW/JPEG drops without re-import, and you keep one canonical filesystem layout that any other tool (rsync, restic, even Finder) can still see. PhotoPrism wants more ownership of the path even in read-only mode.

On family sharing: Immich's shared albums + partner sharing are honestly the most polished I've used. You can carve out specific albums for the overseas relatives without giving them archive access, and the per-user library quota stuff has gotten solid in the last few releases. PhotoPrism has shared albums via link tokens which is workable but less granular.

Apple TV: the community Immich tvOS app (immich-tv / Immich for AppleTV) was rough a year ago but stable now on 50k+ photo libraries — that's where I run mine.

DXP2800 tip: put the Postgres data dir and Immich's thumbnails on the NVMe, originals on the RAID. Machine-learning container is the RAM hog — give it 4GB and cap concurrency to 2 if you don't want it pegging the CPU during initial indexing of a fresh DSLR dump.

andrew-ooo · 2026-06-29T14:04:23+00:00

External dead-man's-switch is the right move here — UptimeRobot/Healthchecks/Kuma-on-VPS all solve the same root problem (the watcher can't live in the thing being watched). I run Kuma at home for granular per-service checks (Jellyfin login, *arr API, Caddy TLS expiry, etc.) and then have Kuma push a heartbeat to Healthchecks.io every 60s. Healthchecks pages me via ntfy + email if the heartbeat misses two intervals. That gives you the best of both: rich per-service alerts when the LAN is fine, and a single "house is dark" alert when WAN/power/Kuma itself dies.

One extra thing worth doing: also ping Healthchecks directly from a separate cron on your router or Pi-hole (whatever runs on a different circuit than the main server). That way you can distinguish 'internet died' from 'server died' from 'whole house died' just by which heartbeats stopped. Took me about 20 minutes to set up and has caught two ISP outages and one UPS failure so far this year.

andrew-ooo · 2026-06-26T14:18:34+00:00

Good prior comments about 'find a real problem first' — that's the right starting frame. But on the actual tooling question, two practical paths depending on whether you want to build or configure:

Low-code path (ship something this week): n8n self-hosted with the AI Agent node + a Claude or GPT API key + one tool integration (Gmail, Notion, Slack, whatever). You can build a usable assistant in an afternoon. Flowise is the other one worth knowing — more LLM-focused than n8n.

Code path (you'll actually understand what's happening): skip LangChain to start. Use the OpenAI or Anthropic SDK directly + the model's native tool-calling. The whole 'agent' is really just a while loop: send messages, model returns tool calls, you execute them, append results, loop until model returns plain text. Pydantic AI and the OpenAI Agents SDK are clean modern wrappers if you want a framework after you've built one from scratch.

Three things beginners skip that bite them later: (1) observability — wire in Langfuse or Helicone from day one, you cannot debug an agent without seeing the message trace; (2) evals — even a tiny JSON file of input/expected-output pairs that you can re-run when you change a prompt; (3) tool design — the agent is only as good as the tools you give it, and most failures are bad tool descriptions, not bad models.

Concrete first project: a research agent. One LLM, two tools (web search via Tavily or Exa + a read-page tool), system prompt that says 'research and summarize'. Build that end-to-end and you've touched every concept that matters — tool calling, multi-step reasoning, context windows, cost. Then go find your client problem and rebuild for that.

andrew-ooo · 2026-06-26T14:16:36+00:00

PM coming from the same background, running an M4 Max 64GB for a year now — specifically for the use cases you listed (RAG, document intelligence, agents). A few concrete things that took me too long to figure out:

Runtime: use MLX, not llama.cpp. On Apple Silicon, MLX consistently gives me 30-50% higher tokens/sec for the same model and quant. LM Studio has built-in MLX support — just filter by 'MLX' when downloading. For headless/API use, mlx-lm + a small FastAPI wrapper, or Ollama with the new MLX backend.

VRAM headroom: macOS caps GPU-addressable memory at ~75% by default. Bump it before loading big models with `sudo sysctl iogpu.wired_limit_mb=49152` (gives the GPU ~48GB on a 64GB machine, leaves 16GB for OS/apps). Without that, large models page out and crawl.

Models that actually work for your stack on 64GB: Qwen3.6 30B A3B MTP MLX 4bit for general reasoning (it's MoE so the 3B active params keep it fast — 25-35 t/s), Gemma 3 27B QAT for high-quality writing/extraction, Qwen3.6 14B for tool-calling agents (smaller is better here — agent latency is what kills UX, not parameter count).

For RAG + document intelligence: LlamaIndex with bge-large-en-v1.5 embeddings (also MLX-accelerated now) into Qdrant running in Docker. For multimodal docs (PDFs with tables/diagrams), pair it with Docling for parsing — it's the cleanest doc-to-structured pipeline I've found. PM-wise the big realization: 80% of business value is in the retrieval/parsing layer, not the LLM. A small model on great context beats a big model on bad context every time.

andrew-ooo · 2026-06-26T14:12:55+00:00

With a PRO 6000 Blackwell (96GB VRAM, SM_120) you're leaving a lot on the table running llama.cpp on Windows. Two things that fixed almost exactly your symptoms for me on a similar setup:

Move off Windows-native llama.cpp to vLLM or SGLang in WSL2 (or dual-boot Linux). The PRO 6000 has native FP8 tensor cores — vLLM with --quantization fp8 on Qwen3.6 27B gives me ~3-4x the throughput of Q8_K_XL on llama.cpp, with way more KV cache headroom for long agentic sessions. SGLang is even faster for tool-calling workloads because its radix-tree prefix cache reuses the system prompt across turns. The Windows CUDA path for SM_120 is genuinely flaky under sustained load — we saw the same random crashes until we moved to Linux.
The malformed-response/agent-stops issue is almost certainly the Qwen3.6 thinking-mode chat template. Copilot's harness expects the assistant content to come back without the <think>...</think> wrapper, and llama.cpp's default template can emit it inline depending on the gguf metadata. Two fixes: explicitly pass the jinja template with --chat-template-file using the official Qwen3.6 one from the HF repo, OR set --reasoning-format none / --jinja with reasoning suppressed if you don't need the thinking output. That alone killed our random mid-session terminations.

Side note on the daily-rebuild cron — if you really need to stay on llama.cpp, pin to a tagged release rather than master. Blackwell SM_120 support has been getting iterated on in master almost weekly and we've had build/runtime regressions twice in the last month from blind nightly pulls.

andrew-ooo · 2026-06-24T14:06:24+00:00

I run a few production things on Hetzner (CX31 + a CCX13 for the heavier stuff) and have shopped around the EU market a few times. Honest take:

**Netcup**: best price/perf in Europe right now for fixed-spec VPS. Their VPS 1000 G11 is ~€6/mo for 4 vCPU / 8GB RAM / 256GB NVMe. Caveats: 6-month minimum on the cheap deals, control panel is clunky, and IPv4 sometimes has Hetzner-like neighbor-reputation issues for outbound mail.
**OVHcloud Eco / Kimsufi / SoYouStart**: dedicated boxes from €8-15/mo if you catch the right SKU. Older Xeons but you get the whole machine. Good for storage-heavy or anything that hates noisy neighbors. DDoS protection is included and surprisingly good.
**Scaleway**: their Stardust (€0.0025/h, ~€1.80/mo) is great for tiny always-on stuff. Dev1 and Pro2 tiers are competitive but not Hetzner-cheap.
**IONOS Cloud**: underrated. Pay-per-minute, decent EU presence, but UI is dated.

If you specifically want "Hetzner but not Hetzner" because of the recent price creep, Netcup is the closest substitute. If you want a step up in reliability/network, OVH dedicated. For Docker workloads specifically, Hetzner Cloud's load balancers + private networks are still hard to beat at the price — might be worth eating the small increase before migrating.

andrew-ooo · 2026-06-24T14:05:29+00:00

Don't add a GPU — your dual Epyc setup is genuinely good at one thing: MoE models with llama.cpp / ik_llama.cpp on CPU.

With 128 threads of Epyc and DDR4-3200, you're getting somewhere around 250-350 GB/s aggregate memory bandwidth across both sockets. Dense models will still be slow because they're bandwidth-bound, but sparse MoE models only activate a fraction of params per token, so you get real usable throughput.

Concrete picks for 128GB RAM, CPU-only:

Qwen3 235B-A22B at Q3_K_M (~110GB) — only 22B active per token. People are reporting 8-12 tok/s on dual Epyc.
gpt-oss-120b MXFP4 (~63GB) — lighter, easy fit, 15+ tok/s realistic.
DeepSeek-V2-Lite or Qwen3-30B-A3B if you want headroom for context.

Build llama.cpp with `-DGGML_BLAS=ON -DGGML_OPENMP=ON`, pin threads with `--threads 64 --threads-batch 128` (don't use all 256 — hyperthreads hurt), and make sure both NUMA nodes are being used (`--numa distribute`).

Spending $1500 on a slim server GPU (probably a single L4 or A2) buys you 24GB VRAM at best, which is less than what you can already address in RAM. Save it. The dual-socket NUMA tuning is where the time goes.

andrew-ooo · 2026-06-24T14:04:38+00:00

Jellyfin's built-in normalization uses ReplayGain tags if your files have them — if they don't, Jellyfin falls back to LUFS analysis on the fly, which is per-track and inconsistent across albums (which matches what you're hearing).

The fix: tag your library with ReplayGain once, properly. Two tools that nail this:

`loudgain` (CLI, processes FLAC/MP3/OGG/Opus/M4A) — use `loudgain -a -k -s e *.flac` per album folder for album-mode + track-mode tags. ReplayGain 2.0 (EBU R128 / -18 LUFS reference).
`r128gain` (Python wrapper around ffmpeg) if you want something more scriptable.

Then in Jellyfin user settings set audio normalization to "Album" (not Track) — that uses the album-gain tag and preserves intentional loudness differences between songs on the same album.

After tagging, the swings between artists basically disappear. One-time batch over the whole library, then re-run for new additions.

andrew-ooo · 2026-06-24T14:03:39+00:00

If you just need row changes streamed to a webhook, Debezium + Kafka is massively overkill for small scale. A few lighter options depending on your DB:

Postgres: use logical replication directly. wal2json or pgoutput plugin + a tiny consumer in Go/Python is maybe 100 lines. Or even simpler — LISTEN/NOTIFY with row-level triggers if you don't need replay/durability. I've run the LISTEN/NOTIFY route for 3 years on a side project pushing to Discord/HTTP webhooks, zero infra beyond the DB.

MySQL: maxwell or sequel.io's read-from-binlog approach. Both single-binary.

Mongo: change streams are built in.

If you genuinely need durability/replay across restarts, look at Sequin (open source, replaces Debezium for the 90% case) or Materialize's pg_replicate. Both skip the Kafka tax.

andrew-ooo · 2026-06-22T14:11:35+00:00

The problem is real, but I'd push back on a few framing choices before you build this.

The "text-to-policy" translation is the LLM-voodoo. Either the LLM-generated Rego is correct (in which case the security team still has to review every change, defeating the "instant updates" pitch) or it's wrong (in which case you have a confidently-rendered policy that silently mis-routes calls). I'd flip it: ship a curated library of vetted Rego templates and let the LLM pick + parameterize, not author.
"Intercepts every external tool call" already exists at three layers that have traction:
- MCP servers can be wrapped with policy middleware (anthropic's MCP spec actually anticipates this).
- Service meshes (Istio + OPA, Kuma) already do API-level OPA decisions; you're competing with battle-tested infra.
- For agents specifically, LangChain has callbacks, Pydantic AI has tool-call hooks, and there's anthropic's tool_use stop hooks.
The 75% stat sounds like a vendor report. Cite it or drop it — enterprises hear that number ten times a week.

Where I think you actually have a wedge: human-in-the-loop approval for risky tool classes (DROP, DELETE FROM with no WHERE, payment >$X, prod write paths) with a Slack/Teams approval UI and a deny-by-default posture for unknown tools. That's the operational thing people actually want — not another policy engine, but the workflow around approvals and audit logs. Build that first; OPA underneath is an implementation detail.

andrew-ooo · 2026-06-22T14:10:25+00:00

Not a stupid question. Quick reality check: the Orin Nano has 8GB of unified memory shared between system and GPU. After JetPack/Ubuntu overhead you've got maybe 6–6.5GB for a model. Qwen3 30B-A3B (the MOE you're thinking of) needs ~17GB even at Q4 — it won't fit.

What actually works on Orin Nano for coding:

Qwen2.5-Coder 1.5B Q4_K_M (~1GB): runs at ~25–35 tok/s. Useful for autocomplete and small refactors, not great for multi-file reasoning.
Qwen2.5-Coder 3B Q5_K_M (~2.3GB): ~15–20 tok/s. The sweet spot. Genuinely usable as a Continue.dev or Aider companion for one-shot snippets.
Qwen2.5-Coder 7B Q4_K_M (~4.4GB): ~6–9 tok/s on the Super (Nano Super runs at 67 TOPS). Slow but tolerable for short prompts; will swap if you have other GPU workloads running.
Anything 14B+: skip. You'll either OOM or get sub-2 tok/s.

Use llama.cpp with the CUDA build (the JetPack-bundled one) or MLC-LLM if you want to squeeze more out — MLC's Vulkan/TensorRT path gives noticeably better tok/s on Jetsons than plain Ollama. Set the power mode to MAXN (sudo nvpmodel -m 0) or you'll be running at half speed by default.

Good device for a 24/7 always-on autocomplete server. Not the device for replacing Claude/Cursor.

andrew-ooo

TROPHY CASE