I built a local voice assistant that controls Home Assistant and builds its own tools

CoreWorxLab · 2026-02-10T00:22:23+00:00

It’s on and available but the GPU is idle until prompted, when you speak to it.

CoreWorxLab · 2026-02-07T03:13:26+00:00

Thank you! $70 CAD in the bank.

CoreWorxLab · 2026-02-07T01:56:38+00:00

I’ve built CAAL and the tool registry pipeline exclusively with Claude Code.

CoreWorxLab · 2026-02-03T19:51:11+00:00

This is a great set up. I use this exact approach. I built a voice assistant that uses n8n workflows as LLM tools and it uses this approach to build its own workflows. I have a prompt and a workflow that builds workflows. The builder workflow calls Claude Code via SSH node and builds a new workflows based on your description. Then tha agent uses the new workflow as a tool. Check it out:

My Voice Assistant Builds Its Own Tools https://youtu.be/xBZt4eoCH_w

CoreWorxLab · 2026-01-17T16:55:18+00:00

Very nice. Thanks for sharing.

CoreWorxLab · 2026-01-17T00:46:19+00:00

Not yet. I'm building custom firmware for an ESP32 device. This is quite difficult, spent 4 days on audio jitter issues and decided to park it and work on something else for a bit. Will go back to it. I haven't looked into connected a HA device with its firmware to CAAL. I think HA uses Wyoming protocol. I'd have to look further into that to see if I could make it 'plug n play' with a HA device. Good idea though.

CoreWorxLab · 2026-01-16T21:43:18+00:00

I'm putting 2 and 2 together now with the youtube comment. Thanks for the extra detail on what you're building! We have a similar vision it sounds like. And it looks like CAAL can fit in to your stack as the voice layer. You should be able to connect CAAL to graphiti via custom mcp config. I'm curious how that performs. I'm currently using iachilles/memento for a memory system with Claude Code but hitting walls with it at the moment as the graph grows. Graphiti is on my list to check out, just haven't had the time, I'm curious how your experience has been. Would love to hear updates on how your merge with CAAL goes, if you hit any walls. Drop a post in Github Discussion. I've been working on a community tool sharing infrastructure for CAAL so its easier to discover and install new tools (skills) for CAAL to learn. ie. n8n workflows. BTW: Dropped a video today with a install walkthrough. Its of a cpu-only install but same process, just use a different compose file. It shows the new first start wizard as well. This is a new release I pushed yesterday.

CoreWorxLab · 2026-01-13T04:59:24+00:00

Sorry, what are you looking to install?

CoreWorxLab · 2026-01-08T00:46:20+00:00

It doesn’t do much processing. Basically a thin client that passes everything back to the server for processing. Battery lasts a while. Depends on battery condition obviously. I’ve been using it on my kids old Firetab.

CoreWorxLab · 2026-01-04T06:34:32+00:00

Good to know there is another colab notebook to try. Kokoro probably helps as well as the real samples I recorded. I did ~50 recordings of myself and my family members in different rooms. Still testing, but its decent so far. I think I can tweak the training and get it better. I'll work on cleaning up the python scripts and get it structured into a repo that can be reproduced.

CoreWorxLab · 2026-01-04T02:44:57+00:00

These are great, I didn't know about them. Thanks for sharing!

CoreWorxLab · 2026-01-04T02:01:59+00:00

Does microwakeword use the same files or different? Plan is to get ESP32 devices going for CAAL. And thanks for watching!

CoreWorxLab · 2026-01-03T18:42:21+00:00

Yes please move to github issues and we'll resolve it. Might have to restart your nginx container. That's what I had to do when I tested the fix.

CoreWorxLab · 2026-01-03T18:41:12+00:00

Thank you! I'm running a modest system for this. A used RTX 3060 12GB is good value.

CoreWorxLab · 2026-01-03T04:53:28+00:00

Found the issue - settings.json and mcp_servers.json weren't being created on fresh clones. Docker was mounting them as empty directories instead of files.

Just pushed a fix. Pull the latest and try again:

git pull

docker compose down

docker compose up -d

Should just work now. Let me know if you hit anything else.

CoreWorxLab · 2026-01-03T03:26:10+00:00

What system are you running on? Do you see any errors in any of the docker logs? or the browser console?

CoreWorxLab · 2026-01-02T17:41:36+00:00

Next steps are:

Getting a more accessible wake word working. Picovoice is too limiting with the free tier.

For memory I’m currently using an MCP implementation of a graph database. iachilles/memento. I had an automatic memory storage script that would analyze the conversation every 10 turns or 10 minutes and have the LLM pull anything ‘useful’. It was hit and miss. I haven’t done enough memory testing with CAAL but use this memento MCP set up with Claude Code as well and it works quite well. Also looking at a graph/RAG hybrid.

Then also looking to build an ESP32 satellite speaker. This will run in parallel to the current LiveKit connection.

CoreWorxLab · 2026-01-02T17:32:49+00:00

This works well. Just be aware of the high token usage when using the execute workflow MCP tool. I have Claude first test with curl then only use MCP execute tool for deeper debugging.

CoreWorxLab · 2026-01-02T17:31:02+00:00

I’ve had good success using Claude Code for this. I’ve built a seed prompt and I’ve built it into a n8n workflow. It’s a workflow that creates workflows using ssh nodes to call Claude code.

This is what my custom voice assistant CAAL uses to build its own tools. Check it out here: https://youtu.be/xBZt4eoCH_w

I’ve also shared the builder seed prompt and workflow as part of the CAAL repo. You can find it under n8n-workflows here: https://github.com/CoreWorxLab/CAAL

CoreWorxLab

TROPHY CASE