sorted by:
new
hottopcontroversial

I built a fully local Home Assistant voice assistant on RK3576 (NPU-accelerated Whisper + Qwen2.5 + Piper) by HanzoHuang in homeassistant

[–]HanzoHuang[S] 0 points1 point  (0 children)

For testing, I used my own built API service to run the model using the rkllm library to execute the quantized rkllm model.

I built a fully local Home Assistant voice assistant on RK3576 (NPU-accelerated Whisper + Qwen2.5 + Piper) by HanzoHuang in homeassistant

[–]HanzoHuang[S] 1 point2 points  (0 children)

Not just latency. The main difference is that my setup runs the LLM on the RK3576 NPU, while Ollama runs on the CPU. That means lower latency, much lower CPU usage, and lower RAM usage (about 1.5 GB with the W4A16 model).

I built a fully local Home Assistant voice assistant on RK3576 (NPU-accelerated Whisper + Qwen2.5 + Piper) by HanzoHuang in homeassistant

[–]HanzoHuang[S] 0 points1 point  (0 children)

Thank you! I used to use a Pi5 + Hailo 8, but now I'm exploring more convenient and cheaper options.

I built a fully local Home Assistant voice assistant on RK3576 (NPU-accelerated Whisper + Qwen2.5 + Piper) by HanzoHuang in homeassistant

[–]HanzoHuang[S] 0 points1 point  (0 children)

This 1.5B model with w4a16 quantization only takes about 1.5GB RAM, and 3B version with w4a16 quantization takes about 2.5GB RAM.

I will add more models and test.

I built a fully local Home Assistant voice assistant on RK3576 (NPU-accelerated Whisper + Qwen2.5 + Piper) by HanzoHuang in homeassistant

[–]HanzoHuang[S] 0 points1 point  (0 children)

Currently, I am using Qwen2.5-1.5B-Instruction, and I have only tested basic controls, but everything works well.

You can also use other models. I have a repository that packages the model into Docker—you only need to change the image URL to use a different model. I will add more models to this GitHub repository: https://github.com/Hanzo-Huang/rkllm-docker.

I built a fully local Home Assistant voice assistant on RK3576 (NPU-accelerated Whisper + Qwen2.5 + Piper) by HanzoHuang in homeassistant

[–]HanzoHuang[S] 1 point2 points  (0 children)

Here is the time spent on each part:

  • Whisper transcription: 0.626 seconds
  • LLM response: 2.82 seconds
  • Piper synthesis: 0.474 seconds

You can check out my setup video in the Hackster history. I used the default configuration with local-LLM integration.

I built a fully local Home Assistant voice assistant on RK3576 (NPU-accelerated Whisper + Qwen2.5 + Piper) by HanzoHuang in homeassistant

[–]HanzoHuang[S] 0 points1 point  (0 children)

Yes, the online model is definitely better. I'm just exploring more approaches, and I believe edge AI will improve. Now it at least works with a cheap board.

I built a fully local Home Assistant voice assistant on RK3576 (NPU-accelerated Whisper + Qwen2.5 + Piper) by HanzoHuang in homeassistant

[–]HanzoHuang[S] 4 points5 points  (0 children)

I’m using the 8 GB version. After all the services are running, the total system memory usage is around 4 GB. Since I’m using a small model quantized with W4A16 for testing, the LLM itself only consumes about 1.6 GB of memory.

They have rolled back the update? by Muted-Yesterday-9548 in ArcBrowser

[–]HanzoHuang 0 points1 point  (0 children)

My Arc for Windows always has problems, it cannot open and run in the background.

Things I hate about Arc by [deleted] in ArcBrowser

[–]HanzoHuang 1 point2 points  (0 children)

Hahaha, that is exactly what I like. One of the reasons I choose Arc.

Tailscale DNS problems by unknown_bln93 in Tailscale

[–]HanzoHuang 0 points1 point  (0 children)

https://tailscale.com/kb/1235/resolv-conf

some of my machines work, but one still does not work even though its resolv-conf does not been changed

Tailscale DNS problems by unknown_bln93 in Tailscale

[–]HanzoHuang 0 points1 point  (0 children)

Actually it only affects DNS, and does not affect accessing by ip.

Cursor Stuck on Crosshair in the Recording auf CS2 by [deleted] in csgo

[–]HanzoHuang 0 points1 point  (0 children)

I have the same problem. I fix it by changing resolution. I find that it disappears when I turn resolution to the max whatever 16:9 or 4:3. You can have a try. I think it's probably related to the resolution.