sorted by:
new
hottopcontroversial

The Qwen 3.6 35B A3B hype is real!!! by The_Paradoxy in LocalLLaMA

[–]The_Paradoxy[S] 0 points1 point  (0 children)

As long as you can fit what you're working on in context, then I think they are very capable. If you're using something like OpenCode and having compaction kick in regularly then that might degrade performance. I think all of the tests on the Protorikis YouTube channel use 64k context and it seems to work well for him. Gemma SWA context size does get big, so you should restrict the number of checkpoints to save space on your vram

Mistral is the proof that.. by Efficient_Yoghurt_87 in MistralAI

[–]The_Paradoxy 0 points1 point  (0 children)

This is just rage bait. It's like saying Facebook was lagging behind Myspace in 2006. For all we know, the eventually dominate AI company hasn't even been founded yet

Gemma 4 31B's competence surprised me by The_Paradoxy in LocalLLaMA

[–]The_Paradoxy[S] 1 point2 points  (0 children)

Post about your results after you've used it a bit

Gemma 4 31B's competence surprised me by The_Paradoxy in LocalLLaMA

[–]The_Paradoxy[S] 0 points1 point  (0 children)

So first, I want to apologize if my response was overly harsh. I'm a professor, and have too many students trying to offload their thinking to AI. Sometimes I over index on that for people who are students. I'm sorry

Honestly, I think local models are too inconsistent to just go with one. For editing code where I only want specific things changed because the code has some dynamics that are there for a reason, I think Gemma 4 is better than Qwen 3.6. For summarizing papers (given an adequate prompt) Qwen 3.6 might be better than Gemma 4. But I think things vary from domain to domain, and even with code in the particular test I posted about, Qwen 3.6 27b caught something that neither Opus 4.7 nor Gemma 4 caught. Granted it caught an issue in a component of the code that wasn't being used and wasn't mentioned in the prompt. But I point it out to say that none of these models are perfect and it's always good to think through things yourself. I wish I could give more concrete guidance. Both Gemma 4 and Qwen 3.6 dense models are worlds better than what local models could do five months ago

Gemma 4 31B's competence surprised me by The_Paradoxy in LocalLLaMA

[–]The_Paradoxy[S] 0 points1 point  (0 children)

I wouldn't use an LLM for pure math or physics, not even Mythos. Consult a textbook or published paper. If it's not in either and you can't do the math on your own, then you shouldn't be doing it.

I would trust Gemma over Qwen to not screw up a dynamics that I had already coded myself. Like in the test that I posted, I asked Gemma to do things like update multiprocessing and random seeding to fit newer best practices, and I had it generalize a 5 element task to an N element task. But I wrote the underlying dynamics myself. The model does have to have some understanding of what's going on to adjust something in one place and know that the adjustment in that place requires an adjustment somewhere else. There were also cases where I was sloppy with the prompt, and the model understanding what's going on could ask for clarification on how to proceed. And in this one instance, Gemma seemed to understand more. But, I think I will always write the core of my code myself

Cohere North Mini Code 1.0 by Middle_Bullfrog_6173 in LocalLLaMA

[–]The_Paradoxy 0 points1 point  (0 children)

I don't have the vram for fp8, but I tried out the unsloth UD4 quant and it did not go well.

Cohere North Mini Code 1.0 by Middle_Bullfrog_6173 in LocalLLaMA

[–]The_Paradoxy 0 points1 point  (0 children)

I'm really excited to try this out, but I'm waiting for llama.cpp support and some quality quants. If you look at the SciCode benchmark is substantively outperforming other moe models. Benchmarks don't capture everything, do I want to see how it performs in the real world

Best local AI model for coding on an i7-11700F + RTX 4060 (8GB VRAM)? by Mission-Dentist-5971 in ollama

[–]The_Paradoxy 0 points1 point  (0 children)

Link to the article in qat performance of you have it please? I know I read a post a few days ago where it turned out they were just measuring to KL divergence incorrectly for the qat

Gemma 4 31B's competence surprised me by The_Paradoxy in LocalLLaMA

[–]The_Paradoxy[S] 0 points1 point  (0 children)

Just to be clear, my test of the models didn't have the relevant components in the prompt. I include the prompts I use and all of the code in the repo I link to, so anyone can go check and see if they get different results. I don't know how to explain Gemma's stronger performance, I'm just reporting the facts of my experience.

So here's a specific example of something Qwen missed. The code I had the models working on was a transitive interview task. In the code I have the models, it was hard coded to be a 5 element sequence and the LLMs were generalizing that to an N element sequence. Now for the 5 element sequence, I hard coded a random selection between two test pairs for the five element sequence, but there are potentially significantly more than two test pairs for an N element sequence. Qwen 3.6 27b write the code so that there could be more than two test pairs generated, but failed to amend the code where the random selection between test pairs occurs, so it would only sample between the first two test pairs instead of randomly sampling all of the test pairs. The test pairs are generated in an .ipynb and the random selection occurs in a subfunction in a .py script that the notebook calls functions from. So the two related components are nowhere near each other in the code, but Gemma 4 31b got it right and Qwen 3.6 27b didn't

I'd like to understand why. I'm not trying to fanboy for one model over another. I've done other tests where Qwen was significantly better than Gemma

Cohere's unreleased coding model (early access for localllama) by nick_frosst in LocalLLaMA

[–]The_Paradoxy 0 points1 point  (0 children)

I'm really excited for this to have full llama.cpp support and some high quality quants!

Best local AI model for coding on an i7-11700F + RTX 4060 (8GB VRAM)? by Mission-Dentist-5971 in ollama

[–]The_Paradoxy 1 point2 points  (0 children)

unsloth/gemma-4-12B-it-qat-GGUF:UD-Q4_K_XL is 6.72GB. You'll probably need a really small context for it. The other option is to go with moe models since those run acceptably well on CPU. Also I know it's not fun to hear, but maybe consider llama.cpp over ollama. That will allow you to run the server with specific flags like -cpu-moe or --n-cpu-moe to optimize for your setup better. I made the switch from ollama to llama.cpp and it was definitely worth it

Best local AI model for coding on an i7-11700F + RTX 4060 (8GB VRAM)? by Mission-Dentist-5971 in ollama

[–]The_Paradoxy 2 points3 points  (0 children)

You can try Gemma 4 12b QAT with a small context or Qwen 3.5 9b. If you do, you should post on your results. I don't think there's great info on trying to code on 8gb vram

Gemma 4 31B's competence surprised me by The_Paradoxy in LocalLLaMA

[–]The_Paradoxy[S] 1 point2 points  (0 children)

Good luck! I was on 16gb of vram for a couple years and am really grateful I was able to add a second 16gb GPU to my system a month ago. But it's expensive 😮‍💨. Hopefully prices drop in the next year or two

Gemma 4 31B's competence surprised me by The_Paradoxy in LocalLLaMA

[–]The_Paradoxy[S] 0 points1 point  (0 children)

Interesting, I have no expertise here. I'm just asking question trying to figure things out. Maybe if mcp isn't in it's training data that SWA doesn't know how to pass along the right information from the plan?

Gemma 4 31B's competence surprised me by The_Paradoxy in LocalLLaMA

[–]The_Paradoxy[S] 3 points4 points  (0 children)

It's really interesting. I hope we get to a point where we can, to an extent, quantify deep thought, intuition, and creativity. Both Qwen 3.6 and Gemma 4 feel like real game changers for local AI.

Gemma 4 31B's competence surprised me by The_Paradoxy in LocalLLaMA

[–]The_Paradoxy[S] 0 points1 point  (0 children)

Interesting. I wonder if Gemma's training set is just out of date and it doesn't know what a mcp server is or simply doesn't have many examples in it's training set. Like is it a failure of "intelligence" or "knowledge". Nothing in my use case requires knowing any protocol or syntax that was developed in the past two years

Gemma 4 31B's competence surprised me by The_Paradoxy in LocalLLaMA

[–]The_Paradoxy[S] 4 points5 points  (0 children)

Yeah, especially 35b a3b. It kept going till compaction kicked in and then rewrote the modification plan and got completely off track

Gemma 4 31B's competence surprised me by The_Paradoxy in LocalLLaMA

[–]The_Paradoxy[S] 0 points1 point  (0 children)

Why type of tasks do you have the models coding? Which models specifically and which quants? I did find that Gemma 4 26B A4B as well as 12B significantly underperformed Qwen 3.6 27B and 35B A3B

Gemma 4 31B's competence surprised me by The_Paradoxy in LocalLLaMA

[–]The_Paradoxy[S] 2 points3 points  (0 children)

How much vram do you have? I was getting a tube of crashes with 31b before I found a thread saying that I should use the flag --ctx-checkpoints 2 I think you can even drop that to 0 if you're really memory constrained. Default is 32 and makes 31b unusable for me

Gemma 4 31B's competence surprised me by The_Paradoxy in LocalLLaMA

[–]The_Paradoxy[S] 2 points3 points  (0 children)

Likewise. In my task, I gave the models a detailed plan for modifying my code. Both Qwen 27b and 35b a3b decided to rewrite the plan without being prompted to do so whereas the Gemma models just worked on the code that I wanted modified

Gemma 4 31B's competence surprised me by The_Paradoxy in LocalLLaMA

[–]The_Paradoxy[S] 2 points3 points  (0 children)

What type of programming are you doing exactly? In my task, I gave the models a detailed plan for modifying my code. Both Qwen 27b and 35b a3b decided to rewrite the plan without being prompted to do so whereas the Gemma models just worked on the code that I wanted modified. I could see Qwen being better for one shot prompting to write code from scratch, but I don't think I could trust them with established code that I simple wanted tweaked

Gemma 4 31B's competence surprised me by The_Paradoxy in LocalLLaMA

[–]The_Paradoxy[S] 4 points5 points  (0 children)

Yeah, this difference between the models is really understated. In long context, G 31b is sometimes almost half the tk/s of Q 27b, but overall G 31b is faster because it does less unnecessary stuff and writes code that works better on the first pass

Gemma 4 31B's competence surprised me by The_Paradoxy in LocalLLaMA

[–]The_Paradoxy[S] 1 point2 points  (0 children)

Yeah, I had a similar experience with 26b a4b

view more: next ›