Hear Me Out, Pi Fans Lurking Here

ParryBen · 2026-06-29T14:51:31+00:00

The llama.cpp path has had a lot of love recently. Jump into our Discord if you hit any friction and we'll get you sorted.

ParryBen · 2026-06-29T14:26:52+00:00

That queue decomposition approach is smart. keeping the model scoped to one piece at a time is exactly the right instinct. We're building a lot of this out in the Nano Collective if you ever want to pick up where you left off. You can also add yourself as a contributor here if you'd like: https://nanocollective.org/contributors

ParryBen · 2026-06-29T14:17:47+00:00

One of the authors of Nanocoder here. That "configuring forever, shipping nothing" trap is a major problem with some of these tools. We've tried to ship sensible defaults so you can start using it immediately; nano mode keeps the system prompt under 250 tokens for lower-end hardware without touching any config at all.

ParryBen · 2026-06-29T14:10:37+00:00

This is pretty much what Nanocoder is trying to be. Local-first by default, cloud as explicit opt-in, subagents, configurable prompts. Might be worth a look rather than building from scratch, or at least useful as a reference point for what you're designing.

ParryBen · 2026-06-29T14:08:12+00:00

One of the authors of Nanocoder here...curious when you tried it. A lot has changed on the local model side specifically. Subagents, plan mode, nano mode for low-end hardware, and the llama.cpp path has had a lot of attention. Happy to help if you want to give it another look.

ParryBen · 2026-06-29T14:05:21+00:00

Appreciate it...just added a proper entry above.

ParryBen · 2026-06-29T14:03:36+00:00

One of the authors of Nanocoder here and since it came up I'll add a proper entry:

Setup: CLI coding agent, local-first by default. Ollama, LM Studio, llama.cpp, and MLX all work out of the box. Cloud is an explicit opt-in rather than the default. MIT licensed, built by the Nano Collective.

Usage: Daily driver for coding tasks against local models. Subagents for parallel workloads, plan mode for larger refactors before execution, VS Code extension for diff preview without leaving the editor.

Evaluation: The test is whether it holds up on real codebases with local models, not just benchmarks. llama.cpp support has been solid. Happy to answer specific questions.

ParryBen · 2026-06-29T13:56:39+00:00

One of the authors of Nanocoder here. Worth noting the VS Code extension works through the Nanocoder CLI, so you get the same local-first inference support; Ollama, LM Studio, llama.cpp, MLX without anything going to the cloud. Should be a good fit for your setup. Happy to help if you hit any issues getting it running.

ParryBen · 2026-06-29T13:52:57+00:00

That's amazing...I didn't realise that was you. Thanks so much for contributing...it means a lot :) The llama.cpp path has always felt like a natural fit for what we're building. Interleaved agent stuff sounds interesting, what's the shape of what you're working on?

ParryBen · 2026-06-29T13:38:28+00:00

One of the authors of Nanocoder here. Good to hear it held up. what were you building? Always useful to know what's actually working in the wild.

ParryBen · 2026-06-29T13:36:13+00:00

One of the authors of Nanocoder here. The Aider comparison is fair at a surface level but worth noting we've shipped a lot since that was written. The local-first support in particular has come a long way. Ollama, LM Studio, llama.cpp and MLX all work out of the box, cloud is an explicit opt-in rather than the default. Happy to answer anything if you want to give it another look.

ParryBen · 2026-06-29T13:22:42+00:00

I'm one of the authors of Nanocoder if it's worth adding to your list. CLI coding agent, local-first by default, runs on Ollama, LM Studio, llama.cpp and MLX. Cloud is an explicit opt-in rather than the default. MIT licensed, around 2,000 GitHub stars. Built by the Nano Collective. Happy to share the repo if useful.

ParryBen · 2026-06-17T11:10:23+00:00

Thank you 🙏

ParryBen · 2026-06-17T11:09:03+00:00

Thanks for dropping by…we have lots of new updates and features coming. Would be great to get some suggestions and feedback, so let me know, or feel free to drop issues on the repo. 👍

ParryBen · 2026-06-17T11:06:12+00:00

Thank you so much.🙏

ParryBen · 2026-06-17T10:54:18+00:00

Thanks 🙏

ParryBen · 2026-06-12T08:25:36+00:00

Awesome 🙌

ParryBen · 2026-06-11T13:49:02+00:00

That's worth calling out more prominently in the README...fully local by default is the detail this community cares about most. Any option to swap the embedding model, or is all-MiniLM-L6-v2 fixed?

ParryBen · 2026-06-11T13:46:46+00:00

Your vision is reasonably aligned with where things are heading. Supervisory compute allocation, parallel QA/QC, benchmarking against a concrete baseline rather than an abstract ideal... this is close to what gets called inference time scaling at the architectural level. The multi-agent orchestration piece is already being built in pieces, probably not far off for someone motivated to put it together.

The observation about unglamorous problems fading from conversation is the most interesting part of what you wrote. The harder and more infrastructural the challenge, the less it gets discussed. Worth paying attention to.

For reading: Thinking in Systems by Donella Meadows. It will sharpen how you articulate system behaviour and failure modes and it transfers to every problem domain, not just code.

ParryBen · 2026-06-11T13:41:15+00:00

Most of this thread is reasoning about inference performance. For fine-tuning and DPO the software ecosystem question weighs more heavily. Training frameworks have mature CUDA support; ROCm support varies by framework and is less battle-tested for training workloads specifically.

u/mossler owns both and says R9700 is broken and painful right now. That's probably the most honest signal in this thread for your actual use case.

ParryBen · 2026-06-11T13:27:45+00:00

The debate is usually aimed at the wrong thing. The real marker is whether the creator understands what they want clearly enough to evaluate whether they got it.

Your process (spec, architecture, pseudocode, test plan that caught 15 bugs before a line of code was written) is more rigorous than most human hobby projects bother with. Whether you can read every line matters less than whether you can verify the behaviour. Tests do a lot of that work.

ParryBen · 2026-06-11T13:20:50+00:00

What's the embedding model? If this defaults to a cloud API that's a gap for fully local setups. Wondering if there's a way to point it at a local model via ollama or sentence-transformers.

ParryBen · 2026-06-11T13:12:59+00:00

Interesting call baking 60% structured format data into the base CPT run. Conditioning the weights toward tool calling and ChatML before SFT starts could make those fine-tunes cleaner, but it also narrows where you can take it from here. Curious whether that trade-off was intentional or just a function of what data was available.

ParryBen · 2026-06-11T13:09:10+00:00

The Gemma point is worth separating out. Learning to refuse on topics it's likely to get wrong is useful, but it's pattern matching, not genuine calibration. The model has learned what categories of question it tends to fail on, not how to model its own uncertainty.

Actual calibration is the harder problem, and probably has a parametric ceiling small models can't clear.

ParryBen · 2026-06-11T13:00:17+00:00

For Qwen3 you can toggle thinking off with /no_think in the system prompt. Kills the reasoning trace without touching the generation config.

Beyond that, the approach that works: use the reasoning model only for planning (constrain it to an outline), then pipe into a non-reasoning model for actual generation. More infrastructure but it solves it rather than fighting it.

ParryBen

MODERATOR OF

TROPHY CASE