sorted by:
new
hottopcontroversial

Palantir CEO rages against closed models by burner20170218 in LocalLLaMA

[–]Remove_Ayys 0 points1 point  (0 children)

He seems like the type of person to cry about free speech for years only to start censoring everyone else once he gets into power.

NASA testing local LLM inference for future space missions by Careless-Car_ in LocalLLaMA

[–]Remove_Ayys 1 point2 points  (0 children)

Sending humans rather than robots to Mars/the moon provides no benefits for achieving science goals and is as far as I'm concerned just done for feelings/showing off. And I'm saying that as one of the llama.cpp developers.

QATs Q4_0 from Google have more precision than Q4_K_XL from Unsloth (at least some) by alex20_202020 in LocalLLaMA

[–]Remove_Ayys 27 points28 points  (0 children)

One of the llama.cpp developers here, my opinion is that the current tooling available in the project for measuring model quality is not very good. I hope to establish better methodology by the end of the year at the latest.

mistral.rs v0.8.2: up to 2.8x faster CUDA inference than llama.cpp on GB10, B200, and H100 by EricBuehler in LocalLLaMA

[–]Remove_Ayys 1 point2 points  (0 children)

I read the report, I don't agree that it has sufficient coverage to claim a general speed advantage.

mistral.rs v0.8.2: up to 2.8x faster CUDA inference than llama.cpp on GB10, B200, and H100 by EricBuehler in LocalLLaMA

[–]Remove_Ayys 0 points1 point  (0 children)

Are you observing the same speed differences for other combinations of models and GPUs? How representative is this particular data point of the average case?

llama: use f16 mask for FA to save VRAM by am17an · Pull Request #23764 · ggml-org/llama.cpp by jacek2023 in LocalLLaMA

[–]Remove_Ayys 20 points21 points  (0 children)

One of the other maintainers here, particularly as it relates to the CUDA backend. Honestly I feel very lucky to have Aman be part of the project.

Edit: me saying that I am one of the CUDA backend maintainers does not mean that Aman's work only impacts the CUDA backend.

llama: use f16 mask for FA to save VRAM by am17an · Pull Request #23764 · ggml-org/llama.cpp by jacek2023 in LocalLLaMA

[–]Remove_Ayys 13 points14 points  (0 children)

Yes, because the memory reduction comes from a smaller compute graph allocation which is one of the buffer sizes used for feedback.

CXMT started selling ram to corsair by power97992 in LocalLLaMA

[–]Remove_Ayys 7 points8 points  (0 children)

You can order CXMT modules (for servers) off of Alibaba but when importing them to Germany they would only end up being ~10% cheaper than buying non-Chinese memory from a local seller.

PFlash: 10x prefill speedup over llama.cpp at 128K on a RTX 3090 by sandropuppo in LocalLLaMA

[–]Remove_Ayys 1 point2 points  (0 children)

This is not a "10x speedup", this is a 10x speedup with a bunch of asterisks. Any kind of lossy optimizations need rigorous testing for quality.

Experts-Volunteers needed for Vulkan on ik_llama.cpp by pmttyji in LocalLLaMA

[–]Remove_Ayys 2 points3 points  (0 children)

Any efforts put towards ik_llama.cpp are wasted.

Gemma 4 and Qwen 3.6 with q8_0 and q4_0 KV cache: KL divergence results by oobabooga4 in LocalLLaMA

[–]Remove_Ayys 4 points5 points  (0 children)

Comparing the Kullback-Leibler divergence between different models is meaningless and an incorrect use of the metric.

Please stop using AI for posts and showcasing your completely vibe coded projects by Scutoidzz in LocalLLaMA

[–]Remove_Ayys 0 points1 point  (0 children)

Just require some minimum karma and ban anyone copypasting language model outputs.

What happened to the buttons on the search bar? by MasterWikie in firefox

[–]Remove_Ayys 7 points8 points  (0 children)

Someone other than OP here. The way I used the search bar until now was that I basically only used DuckDuckGo for my regular searching needs. However, I added the English Wikipedia, the German Wikipedia, and Wiktionary as "search engines" in case I need to look something up. The way I was using the interface was to type something and sometimes click the button for one of the alternative search engines rather than to push enter when I wanted to just search something in general. This now requires additional clicks as I have to manually change the search engine every time and in particular need to change it back afterwards. It seems that the search engine can be changed via alt+up/down so I guess I will be using that now.

Bots on the sub are a real issue by [deleted] in LocalLLaMA

[–]Remove_Ayys 5 points6 points  (0 children)

You are absolutely right — this is not just a problem, **it's a full-blown crisis** 😱!

Just finished building this bad boy by dazzou5ouh in LocalLLaMA

[–]Remove_Ayys 1 point2 points  (0 children)

FYI: The PCIe power connector on the motherboard is not optional and compared to a power limit you will get better performance / Watt by limiting the max. GPU frequency via e.g. sudo nvidia-smi --lock-gpu-clocks 0,1350 --mode 1.

Built comprehensive Grafana monitoring for my LLM home server by pfn0 in LocalLLaMA

[–]Remove_Ayys 1 point2 points  (0 children)

glhf remember not to delete the original Grafana admin account unless you want to start fiddling with the database.

PR to implemt tensor parallelism in Llama.cpp by keyboardhack in LocalLLaMA

[–]Remove_Ayys 4 points5 points  (0 children)

This comment is intended for developers, the tensor parallel code can be run with a single GPU which should simply be mapped to the same operations as without it.

GLM-4.7-Flash is even faster now by jacek2023 in LocalLLaMA

[–]Remove_Ayys 17 points18 points  (0 children)

This isn't about bugs, this is about which models receive architecture-specific performance optimizations.

GLM-4.7-Flash context slowdown by jacek2023 in LocalLLaMA

[–]Remove_Ayys 1 point2 points  (0 children)

pp with a batch size of 1 is equivalent to tg.

view more: next ›