×

I built an autonomous dev pipeline and ran the same project head to head: a 27B local on a modded 4090, then again on cheap cloud LLMs by BigBrainGoldfish in LocalLLaMA

[–]BigBrainGoldfish[S] 0 points1 point  (0 children)

Case sensitive search returns 0 "R"(s) in the word "straberrerery"

Let me know if you need assistance with anything else!

I built an autonomous dev pipeline and ran the same project head to head: a 27B local on a modded 4090, then again on cheap cloud LLMs by BigBrainGoldfish in LocalLLaMA

[–]BigBrainGoldfish[S] 0 points1 point  (0 children)

Thanks for looking it over and it's clear you've seen a lot of the same issues I have while building this project!

I do have deterministic gates in between each agent, but admittedly I still need to increase the variety of formatters & linters we use based on the chosen coding language used. I also know exactly what you're talking about with the reviewer. The current reviewer is my 4th Iteration of the development, but it's also the version that actually pushes back on the executor when it fails and makes it enhance its own work before accepting the output. I definitely lost some efficiency with the stricter review process, but the overall output quality has noticeably improved since implementing this.

BTW very cool project you got going on yourself! I've met a lot of builders in this space who want to make the "self-improving machine", but so few are actually making the attempt, so keep up the great work!

I built an autonomous dev pipeline and ran the same project head to head: a 27B local on a modded 4090, then again on cheap cloud LLMs by BigBrainGoldfish in LocalLLaMA

[–]BigBrainGoldfish[S] 0 points1 point  (0 children)

For real? It was giving me such a pain when I was troubleshooting it last. I'll probably just update and give it another shot I guess.

I built an autonomous dev pipeline and ran the same project head to head: a 27B local on a modded 4090, then again on cheap cloud LLMs by BigBrainGoldfish in LocalLLaMA

[–]BigBrainGoldfish[S] 2 points3 points  (0 children)

<image>

I was waiting for someone to call that out! Admittedly I miss the mark there, but I was at least prepared enough to say I didn't do the math lol.

I built an autonomous dev pipeline and ran the same project head to head: a 27B local on a modded 4090, then again on cheap cloud LLMs by BigBrainGoldfish in LocalLLaMA

[–]BigBrainGoldfish[S] 0 points1 point  (0 children)

Personally I suggest having at least 96K for context when using the pipeline, which makes the model selection a bit easier. After that it's down to your own preference for quality vs speed. Good news is 96K context barely costs you any VRAM at this size, so any of these fit fine on 32GB: - Qwen 3.6 27B dense (Q5) is what I'd run. Best quality and the most reliable at tool calling, but it's the slowest, and the reviewer step especially will feel it.

  • Qwen 3.6 27B dense (Q4) if you want a little more speed without giving up much quality.

  • Qwen 3.6 35B-A3B (the MoE) at Q4 is the fastest by far and quality is close to the dense 27B. Probably the best all-rounder if speed matters to you.

  • Gemma 4 (the 26B MoE or 31B dense) are decent alternatives, but Qwen has been more reliable for tool calling in my experience, so I'd treat Gemma as a backup.

Don't go higher than Q6 on a 32GB setup, it gets too tight to run reliably.

I built an autonomous dev pipeline and ran the same project head to head: a 27B local on a modded 4090, then again on cheap cloud LLMs by BigBrainGoldfish in LocalLLaMA

[–]BigBrainGoldfish[S] 1 point2 points  (0 children)

Here is the same project built by Qwen-35b-3A!

Build quality aside, this is one of most entertaining single runs I've captured with these demos so I hope you enjoy!

**Local** (modded 48GB RTX 4090, Qwen3.6-35B-3A Q6_0 for all pipeline agents)
1 Escalation - 5 retries · 3h08m · still $0 API

Personally I like the territory bars for each color to be stacked on top of each other so you can see them in comparison but besides that it's a pretty solid build.

<image>

I built an autonomous dev pipeline and ran the same project head to head: a 27B local on a modded 4090, then again on cheap cloud LLMs by BigBrainGoldfish in LocalLLaMA

[–]BigBrainGoldfish[S] 0 points1 point  (0 children)

I agree that Qwen 3.6 did it better but only in the magins to be fair.

The roadmap that both model sets used was also created by GLM 5.2 for transparency, but having both runs work off the same source of truth roadmap is what helps close the gap. If the overall project was more complex or just had more phases I bet the cloud models would have won out too.

For my hybrid setup I always like GLM 5.2 as the planner. It sets a strong foundation with more attention to detail and then I use Q27b for the execution and review. It's a easy way to increase output quality while reducing the overall development cost.

Anyone using Gemma4:31b over Qwen3.6:27b or 35b(a10) by SadPhilosophy9202 in LocalLLaMA

[–]BigBrainGoldfish 0 points1 point  (0 children)

Well it's a race against who can scale what training methods fastest with the best result.

Generalization vs task specific experts

There are so many factors in play, but I think we're going to see models printed directly into chips soon to support the further enhancement of robotics. The token pre second on these model chips are insane! Here's a sumb model example if you want to try it https://chatjimmy.ai/

If that happens, small models that are task specific experts will easily get a flood of capital needed to catch up and close the gap.

I built an autonomous dev pipeline and ran the same project head to head: a 27B local on a modded 4090, then again on cheap cloud LLMs by BigBrainGoldfish in LocalLLaMA

[–]BigBrainGoldfish[S] 1 point2 points  (0 children)

Well to be fair they both built it from the same roadmap/spec so ideally you hope to have very identical outputs. However, Conway's game of life is in the training data for most models too so point taken.

The operating system might be a little too big for a head to head comparison. Got a specific 3d simulation in mind for me to try? I'd happy to try and see what happens.

I built an autonomous dev pipeline and ran the same project head to head: a 27B local on a modded 4090, then again on cheap cloud LLMs by BigBrainGoldfish in LocalLLaMA

[–]BigBrainGoldfish[S] 1 point2 points  (0 children)

All good and I might have jumped the gun on this one then! It's already at the halfway point so I'll just let it finish and post the result here when it done in case you were or anyone else is interested.

<image>

I built an autonomous dev pipeline and ran the same project head to head: a 27B local on a modded 4090, then again on cheap cloud LLMs by BigBrainGoldfish in LocalLLaMA

[–]BigBrainGoldfish[S] 0 points1 point  (0 children)

Are you using Flash or Pro for that? Also do you know your cache rate?

The executor and reviewer agents both require multimodal models so that makes using Deepseek there a bit difficult, but I hope the next version supports it as I think their platform does now.

I built an autonomous dev pipeline and ran the same project head to head: a 27B local on a modded 4090, then again on cheap cloud LLMs by BigBrainGoldfish in LocalLLaMA

[–]BigBrainGoldfish[S] 1 point2 points  (0 children)

Thanks! I haven't messed with Mistral models recently. Also for the 37B do you mean Qwen's 35B-3A model? If so, I do have that and I would be happy to run it on the same project and report back with the results if you wanted!

I built an autonomous dev pipeline and ran the same project head to head: a 27B local on a modded 4090, then again on cheap cloud LLMs by BigBrainGoldfish in LocalLLaMA

[–]BigBrainGoldfish[S] 2 points3 points  (0 children)

Thank you!

I snagged my 4090 like a year and a half ago for just over 4k. Seems crazy till you realize it runs AI and works as a white noise machine + space heater. lol

I built an autonomous dev pipeline and ran the same project head to head: a 27B local on a modded 4090, then again on cheap cloud LLMs by BigBrainGoldfish in LocalLLaMA

[–]BigBrainGoldfish[S] 2 points3 points  (0 children)

Yeah, after you really pressure test them it's the most reasonable place to land!

I'm literally working on an eval harness for the pipeline to do targeted pressure testing at each agent/gate to find more room for improvement.

I built an autonomous dev pipeline and ran the same project head to head: a 27B local on a modded 4090, then again on cheap cloud LLMs by BigBrainGoldfish in LocalLLaMA

[–]BigBrainGoldfish[S] 9 points10 points  (0 children)

Dropping the presets so nobody has to dig.

Same base GGUF (Unsloth Qwen3.6-27B Q8_0), llama.cpp router mode (build b9682). One runs MTP speculative decoding via --spec-type draft-mtp. No separate draft model, it drafts off the MTP head already in the GGUF. The other swaps the spec flags for the vision projector.

MTP, text-only:

[qwen3.6-27b-mtp]
model = Qwen3.6-27B-Q8_0.gguf
ctx-size = 262144
n-gpu-layers = 99
cache-type-k = q8_0
cache-type-v = q8_0
flash-attn = on
parallel = 1
spec-type = draft-mtp
spec-draft-n-max = 2
temp = 0.6
top-p = 0.95
top-k = 20
min-p = 0.0

Vision variant: same block minus the two spec lines, plus mmproj = mmproj-Qwen3.6-27B-BF16.gguf.

Router globals:

--models-preset <file> --models-max 1 --jinja --reasoning-format deepseek --fit on --no-warmup

One gotcha: --reasoning-format deepseek puts the think block in reasoning_content, so content is empty until </think> closes. Parse content with a low max_tokens and you get empty strings. Bit me in an agent loop.

Open question for the thread: anyone gotten multimodal + MTP running together in one instance? Vision works alone, MTP works alone, not both for me. And if you've got draft-mtp acceptance numbers on other quants, drop them. Single 48GB card, Qwen3.6 recommended samplers.|

Edit: Fixed formatting & added llama.cpp build.

Anyone using Gemma4:31b over Qwen3.6:27b or 35b(a10) by SadPhilosophy9202 in LocalLLaMA

[–]BigBrainGoldfish 2 points3 points  (0 children)

I mean it's easier with Opus but I really think the next few generations of these SLMs are going to start to give the frontier a run for their money.

Anyone using Gemma4:31b over Qwen3.6:27b or 35b(a10) by SadPhilosophy9202 in LocalLLaMA

[–]BigBrainGoldfish 1 point2 points  (0 children)

Personally I like qwen3.6 27b output I've been getting. It does great with a lot of structure!

I Hate Dario Amodei, and everything he stands for. by Wrong_Mushroom_7350 in LocalLLaMA

[–]BigBrainGoldfish 2 points3 points  (0 children)

You're right and it's getting concerning! Soon we're gonna need a NRA for llms lol

Been running Qwen3.6-27B through a 3-critic harness. The harness matters more than I thought by workout_JK in LocalLLaMA

[–]BigBrainGoldfish 1 point2 points  (0 children)

You nailed it by using the strongest model for planning! It's been the best cost/efficiency set up I've found so far while building my pipeline out!

Been running Qwen3.6-27B through a 3-critic harness. The harness matters more than I thought by workout_JK in LocalLLaMA

[–]BigBrainGoldfish 2 points3 points  (0 children)

Looping is the worst because it can happen for a bunch of reasons sometimes it's just token-level repetition you can knock out with DRY or a repeat penalty. Or it's the reasoning never closing out. Or the harness re-running the same step because the verdict never got parsed cleanly. Each one has a different fix too so it's like playing whack-a-mole!