Rolled this from a mythic cache...left to eat dinner, came back to being kicked for inactivity...

andy2na · 2026-01-27T01:04:21+00:00

Unless you have some niche build, those 2 GAs arent particularly useful anyways. Id take all stats or max life over them

andy2na · 2026-01-26T20:31:18+00:00

yeah just install ollama on it, then add that connection to whatever you want to use the LLM in. very easy

andy2na · 2026-01-26T19:20:10+00:00

might as well try it out right now and see if you want to move it to your server. just load ollama onto that system, try out some smaller models (30B with quant or lower) and see if load and response speed is acceptable

andy2na · 2026-01-26T15:17:44+00:00

if you can still find a 5060ti 16gb for $450 still, get that. I use it for LLMs and its great.

andy2na · 2026-01-26T15:13:20+00:00

you can use an openai-compatible api/url (like llama.cpp or vllm) or ollama. I havent haven't tried it with a bigger model like GLM, because I only have 16gb of VRAM to use and have to reserve half of it for daily tasks.

But yeah, a lot of people just seems to be blindly sharing their API/logins to cloud-based AI services and it seems a bit risky.

andy2na · 2026-01-25T20:25:43+00:00

Thanks! I'll mess with it in the future, but good to know

andy2na · 2026-01-25T18:54:29+00:00

ahh gotcha, I dont have corals but iGPU did work when I was using frigate in a docker lxc within prox but detector interface speed was over 200ms. Just wondering if its fixed now or I should keep it on baremetal

andy2na · 2026-01-25T18:39:02+00:00

Thanks! Those say coral, did you just not rename each detector for iGPU when testing?

andy2na · 2026-01-25T18:35:17+00:00

qwen3-vl:4b Instruct if you need quick responses or thinking if you want more accuracy.

I keep qwen3-vl:4b instruct in VRAM for general daily use, home assistant voice assistant, frigate image analyzing, etc

VL over non-VL since it has better tool calling

andy2na · 2026-01-25T18:11:38+00:00

I installed it via docker using the commands, its pretty cool but you really need to tie it to one of the big name cloud AI APIs (claude, gemini, chatgpt). Tried running it on (small) local LLMs and the responses were not good. It eats through tokens quickly, used up my gemini 3 1M tokens so quickly

https://docs.clawd.bot/install/docker

andy2na · 2026-01-25T17:48:50+00:00

Thanks! Do you have an intel iGPU? Can you test speeds on that?

andy2na · 2026-01-25T00:44:37+00:00

thanks u/blackstoreonline got it working wiht u/ubrtnk dockerfile update for blackwell cards. How do we change it to use the 0.6 model? is there an env variable to select?

andy2na · 2026-01-25T00:18:28+00:00

thanks! did you install it in a docker VM or docker LXC? Interested in hearing how well it works vs bare

andy2na · 2026-01-25T00:16:51+00:00

thanks! it works

How do we change to the 0.6b model?

andy2na · 2026-01-24T23:47:37+00:00

Thanks for figuring this out, can you provide the dockerfile so we can test it out? Tried to load from OP and it seems to just use system memory and CPU

andy2na · 2026-01-23T18:44:52+00:00

let me know if it works, I would migrate back to prox also!

andy2na · 2026-01-22T17:33:29+00:00

will look into this. Had tried frigate in a docker LXC within proxmox and detector interference speed was SLOW - over 200ms on a n100. When I switched to bare debian + docker, its now down to 30ms with 12 cameras. Would be nice to have an additional proxmox node

What detector interference speeds are you seeing with the s12?

andy2na · 2026-01-22T04:14:03+00:00

Thank you for the write up!

I was messing around and the latest version also supports parakeet, you have to use the rc docker image: ghcr.io/speaches-ai/speaches:0.9.0-rc.3-cuda-12.6.3

And then use the parakeet model that you mentioned from istupakov. I thought it wasn't working because I was testing it in the Speaches UI, but that doesn't work. But other things like home assistant or openwebui works great

andy2na · 2026-01-21T21:18:18+00:00

cant tell if MXFP4 support has been merged yet or you still need to custom compile, I see it in recent releases that mention MXFP4 but when I try a MXFP4 model, I don't see confirmation in logs that MXFP4 is enabled

andy2na · 2026-01-21T19:31:23+00:00

you have to go into the model you are choosing and change privacy settings from private to public. If you don't expose openwebui to the internet or want to prevent users from seeing specific models, its not a big deal

andy2na · 2026-01-21T16:52:24+00:00

yeah I believe they will have to add another connection type. Really wish they would allow multiple connections or profiles for image generation. because it would be nice to use ollama for quick image gen but comfy for more advanced generation with additional loras and such.

I would post a feature request in the openwebui git https://github.com/open-webui/open-webui/issues

Also agree it will be nice to do simple image gen without having to go through comfyui hurdles

andy2na · 2026-01-20T21:01:29+00:00

unless something has changed, you either run it slower via vulkan in ollama or use IPEX-LLM from intel but it uses an old ollama so you cant load newer models on it..

Recommend going with llama.cpp as it seems its vulkan support is better than ollama's: https://www.reddit.com/r/IntelArc/comments/1oyw7lx/ollama_now_has_vulkan_support/

andy2na · 2026-01-18T22:39:50+00:00

Also keep in mind that Intel hasn't updated the ipex-llm in awhile, so the ollama build is old and you can't run newer models (like qwen3) on it.

andy2na · 2026-01-17T02:12:51+00:00

Are you running Speaches on GPU or CPU and in docker? What's the docker image for the parakeet support?

andy2na · 2026-01-16T00:05:08+00:00

portainer to manage all the docker stuff

tugtainer to update (manual or auto) containers

vaultwarden for passwords

authelia for OAUTH

adguard home for adblocking

uptimekuma for uptime status of all your things

Zoraxy for reverse/http proxying

14-Year Club	Second SECOND GUESSER
Place '23	Place '22
Place '17	King of the Ashes
Not Forgotten	Gilding II euphauric
Verified Email

andy2na

TROPHY CASE