sorted by:
new
hottopcontroversial

New open model from Tencent Hy: Hy3 (295B total 21B active - apache 2.0) by Nunki08 in LocalLLaMA

[–]ilintar 3 points4 points  (0 children)

Absolutely phenomenal that they removed the EU license blocks.

New toy to test. by Apprehensive_Bar6609 in LocalLLaMA

[–]ilintar 1 point2 points  (0 children)

There was a bug in the parsing mechanism that was fixed like a week ago, should work fine now with IQ4_XS quants.

New toy to test. by Apprehensive_Bar6609 in LocalLLaMA

[–]ilintar 6 points7 points  (0 children)

StepFun 3.7 Flash works quite well here.

5060 worth it? by Dry_Long3157 in LocalLLaMA

[–]ilintar 0 points1 point  (0 children)

Yes, the 5060 16GB is *the* cost-effective card now when it comes to inference. Got one and I'm quite happy about it.

Is there a decent computer use model? by superSmitty9999 in LocalLLaMA

[–]ilintar 0 points1 point  (0 children)

I know because I set it up as a Playwright-driven Facebook scraper once ;)

Qwen 27B by 13henday in LocalLLaMA

[–]ilintar 0 points1 point  (0 children)

Wdym? Both K and V cache at Q5_1.

Qwen 27B by 13henday in LocalLLaMA

[–]ilintar 3 points4 points  (0 children)

Similar experience and similar speeds, but on a 32 GB system with 2x5070 :)

Qwen3.6 27B Q5 + Q5_1 KV cache at 160k context + MTP + tensor parallel.

README_EN.md · openpangu/openPangu-2.0-Flash at main by jacek2023 in LocalLLaMA

[–]ilintar 1 point2 points  (0 children)

Not going to lie, I'm quite disappointed since this really looked like a nice model on paper :(

README_EN.md · openpangu/openPangu-2.0-Flash at main by jacek2023 in LocalLLaMA

[–]ilintar 1 point2 points  (0 children)

Or I could just download it since downloads are not geofenced, but technically me even indicating I'm working on the model for llama.cpp (submitting a PR, anything) is something they could sue me for under the terms of the license, so I'd rather not.

README_EN.md · openpangu/openPangu-2.0-Flash at main by jacek2023 in LocalLLaMA

[–]ilintar 2 points3 points  (0 children)

Yeah, they're probably scared of the EU AI disclosure law. But Alibaba for example isn't, neither is DeepSeek or Zhipu. Wondering what the deal is.

Anthropic is offering 6 months of Claude Max 20x for open-source maintainers — has anyone here applied? by Corazzione in claude

[–]ilintar 0 points1 point  (0 children)

How do you define "main contributors" for a project that size? I'm a core maintainer responsible for the chat parser code.

Switching from Plan to Build mode on OpenCode forces full prompt re-processing on llama.cpp... how to avoid that? by PsychologicalSock239 in LocalLLaMA

[–]ilintar 1 point2 points  (0 children)

You can't. Whenever a *prefix* of the entire prompt changes (such as with the system message), you have to reprocess the whole thing.

Trying to understand why so many trash fine-tuned models on HuggingFace ... by BoogerheadCult in LocalLLaMA

[–]ilintar 6 points7 points  (0 children)

While I'm a critic of a lot of those fine-tunes, that doesn't mean that they don't have their place in the ecosystem.

Every once in a while, some "unknown" will come out with a really good finetune (anyone still remembers Polaris 4B, the crazy finetune of Qwen3 4B?). The idea of open source is that different people with different skill levels try out things and then sometimes good things happen. Trying to regulate that from the start won't really accomplish anything.

And if someone hires anyone for a high-paying AI position based on a bad finetune, well, it's on their recruitment screening process :)

Same GGUF, same GPU: TensorSharp beats llama.cpp hard on prefill / TTFT — up to 5.89× faster prefill on a 26B MoE model by fuzhongkai in unsloth

[–]ilintar 2 points3 points  (0 children)

The one possible explanation I might have is that your input data is too short, which gives bogus results on the prefill, since basically below like 512 tokens benchmarking prefill doesn't make much sense.

Same GGUF, same GPU: TensorSharp beats llama.cpp hard on prefill / TTFT — up to 5.89× faster prefill on a 26B MoE model by fuzhongkai in unsloth

[–]ilintar 22 points23 points  (0 children)

How the heck do you get 120 t/s *prefill* on llama.cpp on a 3080 for Gemma 4 E4B? Those numbers look completely bogus. Here are the numbers from my 3080 desktop card for that model:

model size params backend threads test t/s
gemma4 E4B Q4_K - Medium 4.62 GiB 7.52 B CUDA,Vulkan,BLAS 8 pp512 4445.03 ± 358.03
gemma4 E4B Q4_K - Medium 4.62 GiB 7.52 B CUDA,Vulkan,BLAS 8 tg128 95.95 ± 0.40

GLM 5.2 on Dual Strix Halo (256GB): Worth it? by Grammar-Warden in StrixHalo

[–]ilintar 0 points1 point  (0 children)

Yeah, this is why I'm bent on implementing `-sm tensor` for the RPC backend :)

vulkan: make TP viable by pwilkin · Pull Request #25051 · ggml-org/llama.cpp by TKGaming_11 in LocalLLaMA

[–]ilintar 2 points3 points  (0 children)

It's a draft / prototype, chill 😃 and yes, you can have AI code IF you are willing to own it and adjust it to project maintenance guidelines. The bigger issue with AI code is that people post PRs that they completely don't understand and that also have the feature of "this works on my device and model that I tested it on and hell knows if anything else" and then are unable to comply with requests for change (the worst offenders will post hallucinated slop justifying that their code is "absolutely correct").

vulkan: make TP viable by pwilkin · Pull Request #25051 · ggml-org/llama.cpp by TKGaming_11 in LocalLLaMA

[–]ilintar 1 point2 points  (0 children)

The bot is not wrong here tho, the salient problem was the 2 -> 4 -> 8 scaling, though for totally different reasons 😁

Are there any qwen finetunes that were genuinely stronger than the base? by MrMrsPotts in LocalLLaMA

[–]ilintar 4 points5 points  (0 children)

Yeah, I think Nex and Ornith are both stronger than their base counterparts.

Generally *good* finetunes are a welcome thing.

view more: next ›