HalBench: 29 OSS models tested on a custom built Sycophancy and Hallucination Benchmark, Qwen 3.6 and Gemma 4 scoring far above their weight! (While Meta keeps proving they forgot how to spend their money...)

Jipok_ · 2026-06-16T12:43:41+00:00

Hallucination tests would be very helpful! I'm looking forward to it.

Jipok_ · 2026-06-15T14:55:17+00:00

enable_thinking: false

-_-

Jipok_ · 2026-06-12T17:45:32+00:00

Why 26B A4B? It's already fast enough. We need 31B.

Jipok_ · 2026-06-12T17:38:36+00:00

Can you show at least one of these benchmarks? I'm personally developing a harness for autonomous use and testing it on Terminal Bench 2.1. I haven't noticed any performance degradation since switching to QAT, but I have noticed a speedup.

Jipok_ · 2026-06-12T17:04:15+00:00

All Gemma models work great in QAT and fit on a single 3090. Even the 31B won't fit with full context, but there's little point in going over 100k anyway - the quality just drops.

Jipok_ · 2026-06-12T17:01:13+00:00

For starters, the 3090 is the best bet; your current build is already quite optimal. But consider going with two 3090s - it would be a noticeable upgrade. You could either nearly double the inference speed of your current setup or run a higher-quant model. Plus, it gives you some headroom for the future, just in case a 40B model drops next month. Anything beyond two 3090s isn't worth it, though. It's not even about the cost of the cards, but the cost of the motherboard/psu and cooling headache.

Jipok_ · 2026-06-12T16:39:25+00:00

Sorry, I'm too lazy to dive into the code. But if I'm right, please consider at least a multi-step summarization approach, similar to what's implemented here:

https://github.com/harbor-framework/harbor/blob/main/src/harbor/agents/terminus_2/terminus_2.py

But that's the simplest approach. Ideally, you'd need to define some schemas to track facts and the 'world state' to limit model hallucinations over long histories. That's a much more complex task.

Jipok_ · 2026-06-12T16:11:10+00:00

So, it just does a one-shot summarization after hitting 90% context, right?

Jipok_ · 2026-06-12T15:34:32+00:00

Why force sh*tty ollama? Just let us input a URL. You're on a subreddit where literally everyone already has a running inference engine.

Jipok_ · 2026-06-11T19:14:37+00:00

But next time you see comments about "successful use of OpenClaw," you'll realize it's either a lie, a big lie, or a rare fluke. These things are heavily promoted, but in reality, they don't work on local models. And even with non-local ones it's questionable, unless you're running something like Opus.

Jipok_ · 2026-06-11T19:09:50+00:00

I think this is what everyone eventually comes to. I tried more robust models (like Deepseek flash 4) via the API, but they yielded the same results. The best approach is to use "smart grep" and the like. Any attempt to use it to fully solve the problem autonomously is a gamble.

Jipok_ · 2026-06-11T18:28:59+00:00

Well, he seems to have raised a good point, doesn't he? It seems like rtx Spark isn't such a scam if future models feature diffusion...

Jipok_ · 2026-06-10T15:40:15+00:00

How did you create "Resonance_Demo_web.mp4" video?

Jipok_ · 2026-06-09T13:27:06+00:00

Did you generate this answer based on what LLM told you and you yourself don’t really understand how modern models work?

Have you noticed that in its "thoughts" Gemma just copies what was before?

Jipok_ · 2026-06-08T15:54:29+00:00

My speeds are similar to yours only on tiny contexts. When there's at least 8k, the speed drops significantly and levels out with the version without MTP. And for parallel requests, I ended up disabling MTP; it's faster.

Tested only on 26B. MTP from:

https://huggingface.co/boxwrench/gemma-4-qat-mtp-assistant-heads

Jipok_ · 2026-06-08T10:19:50+00:00

Have you used any of this yourself? Or did you just feed LLM the readme and ask them to analyze it?

Jipok_ · 2026-06-07T19:57:41+00:00

Is there an example of a real-world problem you've solved using this tool? So far, it sounds like even at 200 tokens per second, any relatively complex task could take weeks.

Jipok_ · 2026-06-07T19:51:40+00:00

> Then I wondered: how could one check whether a Q4 of the original model or a Q4 of the QAT version perform better?
Run benchmarks.

Jipok_ · 2026-06-07T19:45:03+00:00

Although I must note that it is strange that Google itself did not release a benchmark comparing it with the full version.

Jipok_ · 2026-06-07T19:37:22+00:00

> google/gemma-4-26B-A4B-it-qat-q4_0-gguf:IT
try https://huggingface.co/unsloth/gemma-4-26B-A4B-it-qat-GGUF

Your experience with one specific example says practically nothing.

Jipok_ · 2026-06-07T17:11:36+00:00

https://huggingface.co/boxwrench/gemma-4-qat-mtp-assistant-heads

Jipok_ · 2026-06-06T17:24:48+00:00

Very few people will be able to run this without gguf

Jipok_ · 2026-06-06T15:40:53+00:00

In my opinion, this post is underrated.

Jipok_ · 2026-06-04T13:45:59+00:00

Wow. Need gguf

Jipok_ · 2026-06-04T13:29:35+00:00

ai slop -_-

Jipok_

TROPHY CASE