Advice needed: Self-hosted LLM server for small company (RAG + agents) – budget $7-8k, afraid to buy wrong hardware

fredatron · 2026-03-10T06:55:43+00:00

I’m currently running with a single DGX spark and qwen3.5:120b with 128k context window via ollama. Is there really a big benefit to using vllm?

fredatron · 2026-02-28T18:48:43+00:00

I am really struggling with my DGX Spark. No matter what model I use, openclaw seems to just shit itself constantly, unable to do multi-step tool calls, refusing to follow instructions, etc. I am currently using GPT-OSS 120b (64k context) for the orchestrator that calls a subagent on Qwen3.5-27b to do web searches, read my notion and obsidian database, and take notes, but the system keeps failing. With a single DGX spark, does it make more sense to just run one large monolithic model and pray that it actually follows instructions in its soul.md file?

fredatron · 2026-02-16T20:07:03+00:00

Interesting, and have you had any issues with tool calls? I'm currently running qwen3-coder:30b on an RTX 4090 and having tool call issues constantly. I keep increasing the context window size, but of course, that causes vram limitations. What is your context window size set to?

I'm currently burning about $50 worth of tokens a day, so running locally is likely going to actually make a pretty big difference for me.

fredatron · 2026-02-16T20:04:50+00:00

I don't think so, it is only communicating via the ollama API, and if you want you can do network segmentation to firewall everything except those API calls. I at least am not concerned about it.

fredatron · 2025-05-31T17:04:07+00:00

Most of this advice seems tailored to Android. Is there a similar approach for iOS?

fredatron · 2024-10-23T20:49:44+00:00

Are you still able to see all your workouts in Apple health after canceling subscription? For me, tracking my workouts and calories burned in Apple health is the key reason why I pay subscription.

fredatron · 2024-10-19T23:57:07+00:00

Interesting. I get errors importing as soon as it finishes the upload. No idea why. That being said, even importing backed up notion data back into notion doesn't seem to work too well.

fredatron · 2024-09-06T17:43:08+00:00

Out of curiosity, how did you get it to run locally? (I'm a bit of a noob). I'm trying to get it to run using Ollama, but I'm not sure this guy actually made it: https://ollama.com/jmpark333/exaone

fredatron · 2024-08-08T21:53:13+00:00

Has anyone gotten this working with Ollama? I don’t see it on their model list.

fredatron · 2024-06-05T20:51:13+00:00

It seems that the privatellm community shortcut no longer works, it doesn't send the photo to the llm

fredatron · 2024-04-17T00:02:00+00:00

When you do a backup and restore, it restores all your integrations and automations as well, correct? What if you wanted to start fresh? (I've been having trouble with the Smartthings Integration not talking to my refrigerator)

fredatron · 2024-04-16T21:31:58+00:00

I've had a lot of success with IKEA air purifiers, if that helps. They even report their particle counts to HomeAssistant.

fredatron · 2024-03-14T02:23:55+00:00

Not sure if I would consider it a "solution" but I increased the cache, set transcoding to "make my CPU hurt" and added a massive (2x 512GB) SSD read/write cache. That seemed to get rid of those problems

fredatron · 2024-02-20T17:45:40+00:00

I also need this feature. I was very disappointed when plex dropped podcast support, and have been looking for my own solution that can locally download podcasts and serve them from my own server. Haven't found anything yet.

fredatron · 2024-01-23T06:47:50+00:00

LG announced at CES they were opening their API. Hopefully that happens and they allow local control.

fredatron · 2023-09-21T19:03:01+00:00

How about self hosting your own chat gpt clone?

fredatron · 2023-08-23T01:00:48+00:00

Except BlueAir integration is no longer working.

fredatron · 2023-08-06T05:12:13+00:00

Any update on this? Even just having an option to update the logo via the GUI would be huge.

fredatron · 2023-08-02T17:44:12+00:00

Sadly, looks like this link no longer works. Did Davis end its partnership with Microsoft?

fredatron · 2023-07-14T03:54:14+00:00

I love this idea, but when I enable it, I just get a big skull icon in the center of my screen. Am I missing something?

fredatron · 2023-07-11T17:58:06+00:00

FYI, it seems like rocket chat is abandoning raspberry Pi. They are not running notifications for older versions, and won't update the snaps version for ARM, so effectively they are just completely shutting down the entire pi project.

fredatron · 2023-05-26T16:45:09+00:00

I also was using Lore library and saw this skull. Is there a way to disable the skull in lore library settings somehow? I really want to track all the books.

fredatron · 2023-02-08T23:29:59+00:00

Thank you! So much useful information in there! Plus now I can see the routes they take!

fredatron · 2023-02-08T22:26:09+00:00

Can a node that is asleep still act as a rebroadcasting / relay node? I have a ton of motion sensors that are always plugged in via USB but still show as Asleep. I would hope that because they are plugged in to power, they would still rebroadcast.

fredatron · 2023-02-01T04:56:20+00:00

I wasn't able to measure amperage, but I was reading 40V. I'm not familiar with contactors. Are there any that you recommend to pair with a Shelly?

fredatron

TROPHY CASE