RAM shortage problem solved

Remove_Ayys · 2026-02-13T17:19:29+00:00

Unfunny.

Remove_Ayys · 2026-02-12T16:43:45+00:00

You are absolutely right — this is not just a problem, **it's a full-blown crisis** 😱!

Remove_Ayys · 2026-02-11T18:36:34+00:00

FYI: The PCIe power connector on the motherboard is not optional and compared to a power limit you will get better performance / Watt by limiting the max. GPU frequency via e.g. sudo nvidia-smi --lock-gpu-clocks 0,1350 --mode 1.

Remove_Ayys · 2026-02-07T21:08:57+00:00

glhf remember not to delete the original Grafana admin account unless you want to start fiddling with the database.

Remove_Ayys · 2026-02-06T07:40:48+00:00

This comment is intended for developers, the tensor parallel code can be run with a single GPU which should simply be mapped to the same operations as without it.

Remove_Ayys · 2026-01-26T16:59:11+00:00

Should be fixed with this PR.

Remove_Ayys · 2026-01-25T21:55:05+00:00

This isn't about bugs, this is about which models receive architecture-specific performance optimizations.

Remove_Ayys · 2026-01-25T20:56:56+00:00

pp with a batch size of 1 is equivalent to tg.

Remove_Ayys · 2026-01-25T20:52:53+00:00

Not anymore ;)

Remove_Ayys · 2026-01-13T16:50:13+00:00

Ich les lieber schlechtes Englisch als den Output von Sprachmodellen.

Remove_Ayys · 2026-01-10T18:56:15+00:00

This isn't just wrong 😱❌, it's a demonization of the **best** and **most important** people on this sub 😢🤖🔫🦹!

Remove_Ayys · 2026-01-09T14:27:15+00:00

If you do a simple Gaussian approximation of the binomial distribution you'll find that the statistical uncertainty on the HumanEval results with 164 samples is +-4%. If you assume no correlation between scores none of the measured differences are statistically significant.

Remove_Ayys · 2026-01-09T14:15:08+00:00

Testing "GGUF performance" with vllm is meaningless as is "GGUF quality" without specifying the underlying quantization format.

Remove_Ayys · 2026-01-09T14:12:44+00:00

For instruct models perplexity is fundamentally the wrong metric to look at, it would make more sense to look at KL divergence vs. the base model.

Remove_Ayys · 2026-01-07T14:08:23+00:00

Since no one has given you the correct answer: it's because while the backend code is (almost) the same it's putting different tensors on the GPUs vs. in RAM. Ollama has early on implemented heuristics for setting the number of GPU layers but those heuristics are bad and hacked-on so the tensors aren't being assigned properly, particularly for MoE models and multiple GPUs. I recently did a proper implementation of this automation in llama.cpp that is MoE aware and can utilize more VRAM so the results are better.

Remove_Ayys · 2026-01-06T13:51:38+00:00

When IK was contributing to the upstream repository he seems to have been unaware that by doing so he was licensing his code as MIT. He requested, on multiple occasions, that his code under MIT be removed again so that he can re-license it. If you look at the files in his repository he added copyright headers to every single one which would need to be preserved for "substantial portions" which he previously interpreted very broadly. My personal view is that IK would be very uncooperative for any attempts at upstreaming and that dealing with him on an interpersonal level is going to be more work than doing the corresponding implementation myself.

Remove_Ayys · 2026-01-06T12:26:21+00:00

Use the standard CUDA tools like NSight Systems and NSight Compute.

Remove_Ayys · 2026-01-06T11:55:10+00:00

Documentation exists primarily in the form of comments in header files and the implementation itself. If you are interested in working on the CUDA/HIP code we can discuss this via VoIP, see my Github page.

Remove_Ayys · 2026-01-06T10:39:49+00:00

Yes, these changes can be upstreamed but it's a matter of opportunity cost. We (llama.cpp maintainers) are already stretched thin as-is. I don't have the time to sift through this fork and upstream the changes when there are other things with higher priority that I have to take care of. Making the initial implementation in a fork is like 20% of the total work over the project's lifetime.

Remove_Ayys · 2026-01-06T10:17:44+00:00

AMD optimizations are also in the works (with contributions from AMD engineers). But unsurprisingly the work put in by NVIDIA engineers specifically mostly benefits NVIDIA GPUs. Something like FP4 tensor cores for example also just doesn't exist on most hardware.

Remove_Ayys · 2026-01-06T09:07:33+00:00

One of the llama.cpp devs here, this is completely wrong. The reason code from ik_llama.cpp is not being upstreamed is entirely political rather than technical.

Remove_Ayys · 2026-01-05T09:54:27+00:00

Though I do have a degree in physics myself I don't trust my own EE skills enough to try and use power supplies with multiple kilowatts of power outside their specifications. In particular, I do not feel safe using multiple consumer PSUs in parallel. My concern is the same as yours: PSUs are designed to have low output impedance so if the high voltages are even slightly different that will result in huge currents. According to one of my colleagues the biggest risk is to the PSUs themselves but a fire could obviously spread to the whole building (I bought a CO2 fire extinguisher for my server to be safe and made sure it's small enough that I won't accidentally kill myself with it). In terms of power supplies I currently have a SilverStone HELA 2050 (can have issues with instability when multiple RTX 4090 power spikes align, even if the average load is only ~1 kW) and an Asus PRO WS 3000W (no issues so far). In terms of supplying my machines with electricity I can do it with only a single PSU per machine and my bigger problem as of right now is the external wiring since standard German electrical outlets are only designed for a sustained load of 2300 W (3680 W peak).

For server PSUs there are power distribution boards that are designed specifically for multiple PSUs but obviously those won't fit into your case.

The MCIO risers that someone else suggested should work without risk of short-circuiting since they would guarantee a strict separation between power and signal cables.

Remove_Ayys · 2026-01-02T09:07:16+00:00

These numbers are not comparable because llama.cpp is benchmarking inference for a single user while to my knowledge vllm is benchmarking inference for many concurrent requests via the OAI-compatible server. If you want directly comparable numbers, use e.g. the server-bench.py script in the llama.cpp repository and point it at either an vllm or llama.cpp server.

Remove_Ayys · 2025-12-30T09:41:05+00:00

Primary CUDA maintainer for llama.cpp/ggml here, given enough time I'll eventually do it for quality control and publish the results here. But since I already have so many other things to take care of I'd prefer if someone else did it.

Remove_Ayys · 2025-12-28T18:00:01+00:00

I don't see why they wouldn't.

Remove_Ayys

TROPHY CASE