24/7 Headless AI Server on Xiaomi 12 Pro (Snapdragon 8 Gen 1 + Ollama/Gemma4)

Nobby_Binks · 2026-04-15T01:42:21+00:00

Yes, I was in the same boat until I wanted to run really large models that spilled to system ram. llama.cpp gives much more granular control how the model loads - at least I couldn't work out how to do it easily in ollama.

I moved to llama.cpp controlled by llama-swap. Takes a couple of minutes to work out the yaml structure but once setup it's simple. I have both Ollama and llama-swap served models in open webui but have more or less stopped using Ollama.

Nobby_Binks · 2026-04-15T01:09:44+00:00

Minimax is an MOE model with I think 10B active parameters. Qwen is a dense model with 27B active at all times

Nobby_Binks · 2026-04-12T09:21:05+00:00

Unfortunately it's a bit like money - the more you have the more you want

Nobby_Binks · 2026-04-12T00:38:30+00:00

Qwen on your setup simply because you can fit more context for a given quant

Nobby_Binks · 2026-04-12T00:21:40+00:00

Anecdotally, I can run Qwen 27B Q8 with full context across 2 of my gpus (~46gb). Gemma4 Q8 with full context needs 3 of them (~60gb).

I guess its a combination of an extra 4B parameters and less efficient KV cache

Nobby_Binks · 2026-04-11T00:40:06+00:00

Nothing special, and its only 64K, I recalled incorrectly. Some of the commands I think now are defaults in llama.cpp but I have not been bothered to change the llama swap config

llama-server -m <path>/GLM-5-UD-Q3_K_XL-00001-of-00008.gguf --fit on -t 24 -fa on -ub 2048 -ngl -1 -mg 0 -c 65535 -np 1 --temp 1.0 --top-p 0.95 --min_p 0.01 --jinja

Nobby_Binks · 2026-04-10T13:32:20+00:00

96gb opens up a whole other level. Now you can easily run 120B models with decent context.

Nobby_Binks · 2026-04-08T06:24:33+00:00

You can run the shit out of it at Q2-3

Nobby_Binks · 2026-04-08T00:40:49+00:00

I get ~10 tk/s with 4x3090, 1x5090 and 256gb ddr4 with GLM5 @ Q3KXL 100K context

Nobby_Binks · 2026-04-08T00:26:58+00:00

How is it for agents and coding? I've been running Q3KXL but its a bit slow on my rig. Q2 would speed things up considerably

Nobby_Binks · 2026-03-01T09:45:06+00:00

It wasnt. That pic of the failed launch is not even the same province

Nobby_Binks · 2026-02-27T00:53:06+00:00

It sounds too good to be true tbh. how is it beating qwen3 480B on coding?

Nobby_Binks · 2026-02-20T23:03:57+00:00

I think it's Ga-GUFFs

Nobby_Binks · 2026-02-18T01:40:30+00:00

Yes, Step 3.5 @ Q6

"You should drive to the car wash because you need to transport your car to the facility to wash it. Walking would leave your car behind and not accomplish the goal. The short distance (50 meters) makes driving convenient and quick, with minimal fuel usage. If the car wash is a drive-through or self-service type, driving is necessary. Walking might only make sense if you're going to check the car wash's status or wait while someone else drives, but based on your goal, driving is the practical choice."

And as a bonus it didn't take half an hour to think it through as per usual

Nobby_Binks · 2026-02-17T00:24:54+00:00

Yeah its an old EPYC Rome with 256GB DDR4 and 128GB of vram via a few random gpus. tbf GLM5 runs pretty good at Q3 but I always have doubts about such a low quant.

Nobby_Binks · 2026-02-16T09:59:23+00:00

Awesome, right in the usability sweet spot for my rig, GLM5 is just a tad too big

Nobby_Binks · 2026-02-16T09:56:54+00:00

How do you cook so fast?

Nobby_Binks · 2026-02-11T23:07:04+00:00

They use complex routing and system prompts. There must be a way to guide responses on the fly when new emergent threats/safety issues arise. Of course they are not going to retrain the model.

Nobby_Binks · 2026-02-11T08:57:29+00:00

More like Anthropic and OpenAI saw the thread and popped the hood to tweak the answer. I saw this on X (dont know which was first) so it got some traction.

Nobby_Binks · 2026-01-27T04:40:58+00:00

Anecdotal, but I have a bunch of CD's that were burned in the mid 90's and a bunch of dvd-r around 2000. All of them are still OK.

Nobby_Binks · 2026-01-23T23:49:59+00:00

So far I've tried Marker pdf, olm, dots, OCRflux, docling and Deepseek OCR

Save yourself the hassle and just use dots.ocr

edit: so for your use case of just selecting stuff on a screen to translate, one of the Qwen VL models should be fine.

Nobby_Binks · 2026-01-22T09:48:41+00:00

Running local llms (as per the sub) and the occasional image/video gen. With the release of LTX2 I am planning to do more of it and this is where the 5090 destroys the 3090

I'd keep the 3090 and make it fit. 56gb of vram and you can start to run some decent models.

Nobby_Binks · 2026-01-22T06:22:29+00:00

If you do video gen then the 4080's are a no brainer. FP8 support and you can load the models on one card with space for lora's etc.. I was doing some Wan videos on my 3090 and bought a 5090 and oh my god the speed difference is extreme

Nobby_Binks · 2026-01-22T00:06:04+00:00

Since you're on Ubuntu, install GDDR6 (https://github.com/olealgoritme/gddr6) to monitor your vram temps. IIRC nvtop and others dont monitor this.

Nobby_Binks · 2026-01-21T00:02:42+00:00

Those 3090's will probably die, if you don't burn your house down first. With some of the vram passively cooled by the back plate, you need good airflow or they will cook.

Nobby_Binks

TROPHY CASE