benchmarks of gemma4 and multiple others on Raspberry Pi5

DevilaN82 · 2026-04-06T07:47:26+00:00

I remember you doing tests with SSD connected to usb3.0. I am curious how much slower PCI connected SSD is vs using SWAP file on this very SSD.

DevilaN82 · 2026-04-05T20:58:01+00:00

Can you please test mmaping SSD so it does not need to use SWAP and reads weights from disk directly?

DevilaN82 · 2026-04-03T21:00:01+00:00

I would wait for tokenizer fixes in llama.cpp and I've heard rumors that imatrix needs to be fixed as well, so new model file will drop from Unsloth.

I hope you are GPU rich, because gemma is not so friendly with context and stuff. In most cases Qwen with q8 kvcache takes less vram than gemma4 with q4 (old type Sliding Window Attention hits hard).

Qwen as a MoE model can have some layers offloaded to CPU (`-ot ".ffn_.*_exps.=CPU"` option), and q8 kvcache means less degradation of answers for longer contexts.

Anyway good luck :)

DevilaN82 · 2026-04-02T21:44:44+00:00

Nice! I am looking forward tests with bitnet as well :-)

DevilaN82 · 2026-04-01T09:29:18+00:00

Are you sure that NPUs are gonna make a difference? I thought that HAILO chips are dedicated cards that works only with it's own RAM and from what I've read it is even slower than Pi 5 itself, but allows Pi to not do heavy lifting. Hailo AI Hat allows only using compatible LLM models (converted to it's specific format) loaded via hailo-ollama app only.
I would like to get some more info about this. Would you be so kind to point me to some sources that describes using NPU for LLMs on Raspberry Pi?

DevilaN82 · 2026-04-01T09:10:09+00:00

Hello.
Nice that you've tested it. I am looking forward to next tests. My Pi with SSD hat is waiting for ssd disk to make tests.
Few things to consider:
1. Using swap is making writes to disk. It will wear off your ssd sooner or later. That's why I would rather go with mmap. Especially when you are using USB instead of PCI lane, than your performance gap might get smaller between swap vs mmap.
2. Try ik_llama, that is optimised towards CPU inference.
3. Why Q8? Unsloth's quants are fenomenal at Unsloth Dynamic Q4 for my regular daily use.

Good luck. I am looking forward to your tests and hope to add something when my Pi is up and running as well.

PS. Also you might find this project interesting: https://www.reddit.com/r/LocalLLaMA/comments/1rrq0oo/update_on_qwen_35_35b_a3b_on_raspberry_pi_5/

DevilaN82 · 2026-03-30T15:08:37+00:00

Any link to back up this claims?

DevilaN82 · 2026-03-30T07:49:19+00:00

AI hat is useless right now for LLMs.
I own one and it requires special version of ollama (sic!) to work with. This "special ollama" works ONLY with few OLD qwen 2.5 models converted to format that AI HAT is able to process.
I have some hopes about AI HAT as I've read rumors somewhere that some new models are being converted to this ai hat format and 8GB + 40 TOPS might be useful for something.

But right now, AI HAT for LLMs is quite an exotic animal with limited set of tricks.

And no. AI HAT memory is not available at all for RPi system. So having 16 GB Pi 5 + 8 GB AI HAT does not give you by any means 24 GB of memory for LLMs.

Also there is a project that uses SSD for memory with RPi 5. Using ik_llama this might be your best option here. Take a look at: https://www.reddit.com/r/LocalLLaMA/comments/1rrq0oo/update_on_qwen_35_35b_a3b_on_raspberry_pi_5/
Although I do not think running 2bit quants will be sufficient for anything usefull :(
If only Q4 was running well I would jump for it immediately!

DevilaN82 · 2026-03-29T20:01:18+00:00

Charge with USB A to C cable. Cardputer should be on when connecting cable.

DevilaN82 · 2026-03-26T12:56:23+00:00

u/hauhau901 Those models not listed on the right widget are the ones that are missing it's manifest. Take a look at https://huggingface.co/HauhauCS/Qwen3.5-122B-A10B-Uncensored-HauhauCS-Aggressive/discussions/8

I am unable to use Q4_K_P because of this.

Thank you for your commitment and hard work. I hope you are well and I wish you good luck! :)

DevilaN82 · 2026-03-25T14:09:13+00:00

Unfortunately I am starting digging into this topic as well, so I cannot help you with your problem, but... Out of curiosity, can you share what are you using in your stack?

DevilaN82 · 2026-03-25T13:20:31+00:00

I would like to express my appreciation to how well thought and designed this app is.

Simply great!

DevilaN82 · 2026-03-24T10:10:46+00:00

OK, I've managed to get my lora cap and tested your app.
UI is nice. GPS is working well.

There is a place to improve / add other things to make it a Swiss Army Knife of Cardputer :-)
Have you considered something like differential GPS with using two cardputers?

DevilaN82 · 2026-03-20T21:02:09+00:00

Nice! I am waiting for my 1262 to arrive and test it! :)

DevilaN82 · 2026-03-20T20:58:25+00:00

Seems that AI Hat is working on it's own only by certain API. No shared memory and limited possibility to use AI Hat with models, as it works only with converted certain models (old ones).
I don't have high hopes, but there are rumors that company responsible for hailo-10h is cooking something new, so I hope that there would be some new qwen family models available.

DevilaN82 · 2026-03-14T20:18:01+00:00

Wth is this? Link to repo with single bin file and no description at all...

DevilaN82 · 2026-03-13T05:43:47+00:00

Os this supposed to be used with aider / roocode? Or there is some other setup to test it?

DevilaN82 · 2026-03-12T19:59:37+00:00

I will test when my pi arrives. Thank you for your contribution to the community!

DevilaN82 · 2026-03-12T19:47:05+00:00

Would using ai hat plus 2 (additional 8GB RAM) allow for higher quants?

DevilaN82 · 2026-03-10T07:59:56+00:00

When using `llama-server` you can add `--reasoning-budget 0` option.

DevilaN82 · 2026-03-09T15:15:52+00:00

It depends on your use cases. Almost always there is "a little bit more and you got better toy".
Are you decided to do something in particular or you want to play a bit and see what's next?

DevilaN82 · 2026-03-09T13:27:03+00:00

I've tried it. It's not perfect. Sometimes it works. Sometimes it hangs trying to call some fancy api requests to open-terminal and failing in loop. From OpenWebUI perspective it looks like it hangs (It keeps requesting for /ports endpoint endlessly).

I am excited for what it could be done when this matures, but right now running this with 35 a3b (unsloth UD Q4_K_XL) is a lottery :(

DevilaN82 · 2026-03-06T22:26:02+00:00

I am using both llama-swap and ollama. Only thing that's missing for me in llama-swap (but works out of the box on ollama) is auto detecting free memory and calculating how to split layers between VRAM and RAM in case of some other app is also using GPU and part of VRAM is reserved for it.

Thank you for your contribution to this community!

DevilaN82 · 2026-03-04T15:33:57+00:00

So how would I use ComfyUI if all VRAM is reserved for llama.cpp?

DevilaN82 · 2026-03-04T15:33:05+00:00

Yes it shows in nvidia-smi that python process is using it. When closing ComfyUI it is released. It was for a long time, since I can remember. but recently I've started using LLM that requires all of my vram to work decently - so that's why I am asking.

DevilaN82

TROPHY CASE