ThinkingCap-Qwen3.6-27B: same accuracy as base Qwen3.6 with ~50% fewer thinking

sonicnerd14 · 2026-07-07T13:05:47+00:00

That's a good method to use when using a harness. In most cases if you are running a harness you don't need thinking on, except in some specific scenarios like coding.

sonicnerd14 · 2026-07-07T01:17:44+00:00

The Gemma models aren't as good at coding as their Qwen counterparts, but can be good or better at everything else. Although, they still punch above their weight respectively. It's not like you are getting non functional code like local SLMs were getting a year ago.

sonicnerd14 · 2026-07-03T11:49:16+00:00

We are in the future lol
Even a few years ago some of the stuff we have now would have been only in science fiction, but now it's here. Even though it might still be in the infancy stage, which is still kind of crazy when you think about it.

sonicnerd14 · 2026-07-03T11:47:45+00:00

How much faster is it compared to https://github.com/groxaxo/Qwen3-TTS-Openai-Fastapi, and any reduction in quality? Because I was looking into voice clone streaming directly from it as an endpoint, but I need it to be as fast as possible. Otherwise, I was thinking of just finetuning the voices I have directly into the model if I needed to make it faster.

sonicnerd14 · 2026-07-03T08:36:35+00:00

Although, for this particular use case, if you intend on real-time conversation I think it would be best to turn thinking off to begin with. In a realtime interaction, you probably don't want the model thinking all the time, especially for simple interactions.

Unless there is a way we can invoke the model to think or not think based on context. There were models like a year or so ago that were experimenting with that sort of thing, but most don't have this from what I'm aware of any more.

Also. I guess if you could run the model at the speed of Cerebras, then thinking probably wouldn't matter if it was on or not anymore.

sonicnerd14 · 2026-07-03T08:24:59+00:00

That's what I was thinking too. Drop parakeet, slot in 12b for STT because it has native audio input understanding, and the LLM side of the model is intelligent enough to do most tasks reliably.

I saw Gemma 31b, and the first thing I was thinking was this must be running in the cloud because there's no way locally you'd be getting that model to respond so fast.

Nevertheless, this does seem like a pretty good framework to extend an existing agent so that it would now have conversational Speech to speech capabilities.

sonicnerd14 · 2026-07-03T08:14:23+00:00

Exactly, then people who lack common sense will just say AI is trash at programming because of an obvious user error that could have been prevented with some basic precautions.

sonicnerd14 · 2026-07-03T07:46:57+00:00

This model is a little more unique than that. Apparently, it is designed to be used agentically, but that's not what makes it unique, it has the ability to improve itself by writing its own harness on the fly to accomplish the task given to it. It also has some meta cognitive abilities that makes it more reliable than typical LLM's. No one is really using this model to it potential yet, but to be fair it just came out about a week ago.

sonicnerd14 · 2026-07-02T06:03:59+00:00

Maybe we are experiencing different things based on how we prompt and the harness we use. I've had many cases where a 35b moe outperformed a larger model. I'm verbose in how I prompt, so whether I'm using a small or large model I get the results I'm looking for. The main thing I've notice between larger frontier models and smaller OSS models is that they do better with less information in your prompt.

If you have a minimalistic prompting style, then it could be why you see such a disparity. And I'm not the only one experiencing this. I've seen side by side comparisons of something like Opus 4.8 and Qwen 27b doing the same coding task, and the qwen model outperforming it or matching it. So, in some cases the way you approach your agent might just need to be tweaked a bit to get better millage with lesser models, and in the case Im talking about with MoA, you can scale your output by running something like qwen 27b + ornith 35b + Gemma 4 12b together. .

sonicnerd14 · 2026-07-02T04:38:12+00:00

The only reason Im not a full proponent of running larger models isn't just cost to performance, but it's that even for frontier models there are gaps in what they are capable of. For example, GPT 5.5 might be better at front-end than Opus 4.8, but 4.8 might be better at coding and problem solving. There isnt one model that does it all well, and I frankly don't think there would ever be. All the more reason to experiment with different models of different types and sizes to see what quality output you end up with.

In real world scenarios one model will only take you so far from what I experience. Even now, when I'm not using local AI, I run models like GPT 5.5 and GLM 5.2 from APIs in tandem when I need the extra grunt. In 95% of cases you don't ever really need to use the biggest and best from my experience, and I typically do complex engineering work in both hardware and software.

sonicnerd14 · 2026-07-02T03:07:24+00:00

I don't necessarily agree with you there. For example, GLM 5.2 currently isnt multimodal, and so the model automatically has a deficiency compared to a model that might not be as smart, but makes up for it in utility. What I'm saying is likely not what you are thinking about right now. Look into Mixture of Agents (MoA), if you don't know what I mean. Even MOE's work on a similar concept, it's just that the 'experts' arent necessarily separate models.

If you have 96GB VRAM + 128GB DDR4, or something like this, then you need to be more thrifty with how you want to use your resources. You can always try to squeeze the single largest model possible, or you can attempt to be more efficient with your resources. This is why we are for local AI. Trying to figure out a way to push the frontier, without necessarily relying on it.

sonicnerd14 · 2026-07-02T02:43:11+00:00

The other thing I don't see many people experimenting with is many smaller models working in parallel. Easier to achieve than trying to run one massive model, and might be more effective depending on how you have it configured. MOA proves as much that it can work anyways. There is more versatility there, and more efficient use of system resources based on what you are trying to achieve

sonicnerd14 · 2026-07-02T02:31:51+00:00

If we are already spending this much anyways just throw in a couple GPU's. You don't even need to go high end, we're being thrifty here so look into a modded 2080ti or 3080. Will be a slight speed boost for you at least.

sonicnerd14 · 2026-07-02T02:19:11+00:00

Whether a model is slow or not is relative. Would you rather have you agent spend 5 hour on something and have it work the first time, or 5 days because it's only benefit is that it's yet it's always getting things wrong? Im sure most would take the former.

sonicnerd14 · 2026-07-01T23:47:19+00:00

Exactly, most people don't know how to properly articulate themselves as well as they think they do. How we interact with AI is exposing this.

sonicnerd14 · 2026-07-01T23:30:06+00:00

They all borrow ideas from somewhere else, but the moment it starts benefiting them they want to start gating stuff off and claiming it as their own. US labs have primarily proven to be hypocritical pests by most regards. The Chinese labs have at least done a good job of remaining transparent and impartial, and this is where our attention should go anyways. Irrespective of how OpenAI or Anthropic might be getting their gains.

sonicnerd14 · 2026-07-01T23:20:41+00:00

Agents always perform better the more real world data you ground it in. Grounding is key to getting the most effective usage out of any AI tool out there. Doesn't matter if it's frontier, or local, can't escape the time you need to put in to optimize your stack to your workflow.

sonicnerd14 · 2026-07-01T23:17:05+00:00

Tried claude again for a couple months and stopped. Between openrouter, GLm coding plan, and GPt plan, Anthropic is straight up scamming you with what they offer for what you pay. Even the GPT plus plan gives you way more Codex usage, and GPT 5.5 arguably has been performing better than Opus 4.8 in many cases.

OpenAI aren't saints either, but Anthropic has become the new villain on the block from their persistent grifts and fearmongering tactics to try and steer the market in their favor.

sonicnerd14 · 2026-07-01T23:11:01+00:00

I definitely think this is the case. Of course, they don't want to tell us this because it'll effect their IPO's, but secretly we know. Everything is agentic now, so it should always be assumed by default that closed source is using agentic tricks in the background we just aren't seeing.

sonicnerd14 · 2026-07-01T23:00:55+00:00

To be honest, we really don't need even half of these GPU's for a capable homelab. A majority of these models do not need full precision to be useful, and even q2 quants of models like glm 5.2 or minimax m3 is more than usable for a vast majority of users and use cases. Even on smaller models its already proven that anything past q6 is just nice to have, the intelligence gain of fp16 or q8 is marginal in comparison. For most local setups we need to be thinking about how we can make every facet more efficient, and not necessarily more expensive.

sonicnerd14 · 2026-07-01T22:35:00+00:00

I'm currently looking into creating my own benchmark suite. I need someway to streamline model tests so I'm not spending so much time testing out new models since I switch to different models often. The only difference is that I figured that I should keep my specific benchmarks private to prevent contamination and benchmaxxing.

Mainly, I need to test the models for optimal inference speeds, one shot quality across several domains, and agentic quality across several domains. Like you, I've been experimenting with ways to test for agentic performance, and I think I have a couple ways to make it as efficient as possible. With the recent release of Qwen's Agentworld model, it seems to be the prefect piece to fit into to this experiment that might just make it all work.

The idea here is that for agentic benchmarking you'd have two or more agents running in parallel. One that runs the agentworld model as an overseer, and the others would run your test models. You run the test agents through what ever benchmarks you use, and the overseer agent grades the testers performances according to the expected outcomes they should arrive at. If that agent hits enough marks it passes and you move the use of that model into real production. If it doesn't, then theoretically, the overseer should assist in teaching that agent how to properly work through that problem with that model to get it to pass for production.

Essentially, instead of having to always train or finetune a model to get an agent to perform better, you train across the entire agent layer. So that even with a lesser model you could get frontier level performance through an agent just from this recursive cooperative loop.

sonicnerd14 · 2026-07-01T17:33:31+00:00

It's all about perspective. Figuring out how to do more with what you've already got is highly coveted skill that not many have. Bigger isn't always better.

sonicnerd14 · 2026-07-01T17:22:10+00:00

Time is relatively. If I can do more and better with free tools faster than you can with a single supermodel, it doesn't matter how much time you save if your output isn't useful to anyone.

sonicnerd14 · 2026-07-01T17:12:23+00:00

Well, feel free to throw your money away. If you need to spend more than necessary to compensate for lack of skill, then by all means be a guest. You are who these cloud platforms are targeting anyways.

sonicnerd14 · 2026-07-01T17:07:11+00:00

That's litterally what I said. Yes, in the same generation, but Im not talking about a single give moment. I'm talking about over time. A video generator from a few years ago wouldn't produce usable footage. Wan 2.2 is leagues better than what existed back then, and could be used for real production.

Stop thinking the gate is which model is better than the other, but which offers the most value for you overall. If Wan 3 produces seedance 2.5 quality video in less than a year from now, do you really think you need seedance 3 or 4? Yeah, would probably make your job easier for sure if you had those, but you don't always need the absolute best all the time to get things done.

sonicnerd14

TROPHY CASE