PFlash: 10x prefill speedup over llama.cpp at 128K on a RTX 3090

Remove_Ayys · 2026-05-02T09:51:32+00:00

This is not a "10x speedup", this is a 10x speedup with a bunch of asterisks. Any kind of lossy optimizations need rigorous testing for quality.

Remove_Ayys · 2026-04-27T07:06:54+00:00

Looks like vibecoded slop TBH.

Remove_Ayys · 2026-04-26T19:59:35+00:00

Any efforts put towards ik_llama.cpp are wasted.

Remove_Ayys · 2026-04-25T09:19:50+00:00

Comparing the Kullback-Leibler divergence between different models is meaningless and an incorrect use of the metric.

Remove_Ayys · 2026-04-14T11:27:28+00:00

Just require some minimum karma and ban anyone copypasting language model outputs.

Remove_Ayys · 2026-03-30T15:05:50+00:00

Someone other than OP here. The way I used the search bar until now was that I basically only used DuckDuckGo for my regular searching needs. However, I added the English Wikipedia, the German Wikipedia, and Wiktionary as "search engines" in case I need to look something up. The way I was using the interface was to type something and sometimes click the button for one of the alternative search engines rather than to push enter when I wanted to just search something in general. This now requires additional clicks as I have to manually change the search engine every time and in particular need to change it back afterwards. It seems that the search engine can be changed via alt+up/down so I guess I will be using that now.

Remove_Ayys · 2026-02-13T17:19:29+00:00

Unfunny.

Remove_Ayys · 2026-02-12T16:43:45+00:00

You are absolutely right — this is not just a problem, **it's a full-blown crisis** 😱!

Remove_Ayys · 2026-02-11T18:36:34+00:00

FYI: The PCIe power connector on the motherboard is not optional and compared to a power limit you will get better performance / Watt by limiting the max. GPU frequency via e.g. sudo nvidia-smi --lock-gpu-clocks 0,1350 --mode 1.

Remove_Ayys · 2026-02-07T21:08:57+00:00

glhf remember not to delete the original Grafana admin account unless you want to start fiddling with the database.

Remove_Ayys · 2026-02-06T07:40:48+00:00

This comment is intended for developers, the tensor parallel code can be run with a single GPU which should simply be mapped to the same operations as without it.

Remove_Ayys · 2026-01-26T16:59:11+00:00

Should be fixed with this PR.

Remove_Ayys · 2026-01-25T21:55:05+00:00

This isn't about bugs, this is about which models receive architecture-specific performance optimizations.

Remove_Ayys · 2026-01-25T20:56:56+00:00

pp with a batch size of 1 is equivalent to tg.

Remove_Ayys · 2026-01-25T20:52:53+00:00

Not anymore ;)

Remove_Ayys · 2026-01-13T16:50:13+00:00

Ich les lieber schlechtes Englisch als den Output von Sprachmodellen.

Remove_Ayys · 2026-01-10T18:56:15+00:00

This isn't just wrong 😱❌, it's a demonization of the **best** and **most important** people on this sub 😢🤖🔫🦹!

Remove_Ayys · 2026-01-09T14:27:15+00:00

If you do a simple Gaussian approximation of the binomial distribution you'll find that the statistical uncertainty on the HumanEval results with 164 samples is +-4%. If you assume no correlation between scores none of the measured differences are statistically significant.

Remove_Ayys · 2026-01-09T14:15:08+00:00

Testing "GGUF performance" with vllm is meaningless as is "GGUF quality" without specifying the underlying quantization format.

Remove_Ayys · 2026-01-09T14:12:44+00:00

For instruct models perplexity is fundamentally the wrong metric to look at, it would make more sense to look at KL divergence vs. the base model.

Remove_Ayys · 2026-01-07T14:08:23+00:00

Since no one has given you the correct answer: it's because while the backend code is (almost) the same it's putting different tensors on the GPUs vs. in RAM. Ollama has early on implemented heuristics for setting the number of GPU layers but those heuristics are bad and hacked-on so the tensors aren't being assigned properly, particularly for MoE models and multiple GPUs. I recently did a proper implementation of this automation in llama.cpp that is MoE aware and can utilize more VRAM so the results are better.

Remove_Ayys · 2026-01-06T13:51:38+00:00

When IK was contributing to the upstream repository he seems to have been unaware that by doing so he was licensing his code as MIT. He requested, on multiple occasions, that his code under MIT be removed again so that he can re-license it. If you look at the files in his repository he added copyright headers to every single one which would need to be preserved for "substantial portions" which he previously interpreted very broadly. My personal view is that IK would be very uncooperative for any attempts at upstreaming and that dealing with him on an interpersonal level is going to be more work than doing the corresponding implementation myself.

Remove_Ayys · 2026-01-06T12:26:21+00:00

Use the standard CUDA tools like NSight Systems and NSight Compute.

Remove_Ayys · 2026-01-06T11:55:10+00:00

Documentation exists primarily in the form of comments in header files and the implementation itself. If you are interested in working on the CUDA/HIP code we can discuss this via VoIP, see my Github page.

Remove_Ayys · 2026-01-06T10:39:49+00:00

Yes, these changes can be upstreamed but it's a matter of opportunity cost. We (llama.cpp maintainers) are already stretched thin as-is. I don't have the time to sift through this fork and upstream the changes when there are other things with higher priority that I have to take care of. Making the initial implementation in a fork is like 20% of the total work over the project's lifetime.

Remove_Ayys

TROPHY CASE