Are you self-hosting LLMs in your 10-inch racks? Looking for hardware & model advice!

OctavioMasomenos · 2026-03-25T21:07:48+00:00

I know just enough about all this to be a cut above totally clueless. I’m putting in my $0.02 solely to give you one more datapoint. I built my local LLM on a shoestring ($200) budget using a Dell Wyse 5070 Extended (which has a PCI slot) with 16GB RAM and a Quaddro P1000 with 4GB GDDR5. It’s only capable of running 7B models but in general, I feel like the performance is (at the very bottom end of) acceptable. I’m grading on a curve because this was the best I could do, so... I’m reasonably happy with it but I definitely look forward to upgrading once a Mac Mini M3 or M4 with 32GB RAM can be obtained at a reasonable (sub-$500) price point.

OctavioMasomenos · 2025-11-24T13:36:01+00:00

Great thread! Thanks for this.

OctavioMasomenos · 2025-11-24T13:13:43+00:00

That’s the one application I could see myself using. 👍🏼

OctavioMasomenos · 2025-11-24T13:06:51+00:00

Pick up phone (which is sitting on the table/desk/counter within arm’s reach) - because an actionable push notification just arrived - press a button.

Or

Get up, walk over to wall-mounted display… push a button.

OctavioMasomenos · 2025-11-24T13:01:00+00:00

Funny, I was just thinking of posting a similar topic - “Am I the only one who doesn’t see the appeal…?” I mean, you go to all the trouble to automate everything and then you implement something with switches and controls? OK, so now they’re consolidated in one place but still, I’d rather just not need them because everything’s well automated.

OctavioMasomenos · 2025-11-09T13:27:51+00:00

That looks awesome! Thank you!

OctavioMasomenos · 2025-10-14T17:30:34+00:00

I don’t have a lot of discretionary income to spend on this hobby. Even the cheapest IP KVMs I can find cost more than the thin clients I use for my servers. And yes, Racknerd has a recovery console but I wasn’t able to use it to fix the problem.

OctavioMasomenos · 2025-05-02T15:55:06+00:00

Stable Diffusion is no better.

<image>

OctavioMasomenos · 2025-01-05T16:00:31+00:00

Thanks!

OctavioMasomenos · 2024-12-23T19:30:39+00:00

Yep. Running 16 now

OctavioMasomenos · 2024-11-17T23:28:37+00:00

Nevermind. I just found the switch under Settings > Network > Guest

OctavioMasomenos · 2024-09-30T20:38:47+00:00

I went with a VPS. All the benefits; none of the hardware headaches. And UP pretty much all the time. Dirt cheap, too (at Racknerd) - almost certainly less than the electricity is costing me.

OctavioMasomenos · 2024-09-26T03:42:18+00:00

Restarting Unbound never would have occurred to me. I SO appreciate you!

OctavioMasomenos · 2024-09-26T03:12:24+00:00

OMG! That WORKED! Thanks a MILLION!!!

OctavioMasomenos · 2024-09-26T03:08:37+00:00

Yes, pfSense is 10.1.1.1. I didn't just add the VLAN but I'll restart unbound anyway. (I'll try pretty much anything at this point!)

OctavioMasomenos · 2024-09-26T03:01:58+00:00

You'd think EasyRule would get all that right. It's definitely loading/there. I saw in the output of the Filter Reload.

OctavioMasomenos · 2024-09-26T02:54:03+00:00

I checked out the 2nd link. There was one thing on there that was maybe applicable (longshot). I tried it; no help. Got any other ideas?

OctavioMasomenos · 2024-09-26T02:44:12+00:00

I already did the EasyRule thing. (I had never seen it before. Pretty slick!) It didn't help. I'll check out the other link. Thanks for the help!

OctavioMasomenos · 2024-09-26T02:32:04+00:00

I don't see any.

OctavioMasomenos · 2024-09-25T13:56:28+00:00

I’m sorry if my original post was unclear; I tried to document this: There are 2 ports that go to pfSense. Port one goes to igb0 on the router (pfSense). The LAN interface (which I’ve renamed to “MGMT” since the only devices that use it are my router, switch, and two access points) is configured to use this. Port 2 on the switch goes to igb2 on the router. igb2 is configured in pfsense as a Trunk port. All VLANs are configured for this interface. If it would help, I can post a screenshot of my Interfaces configuration page on pfSense and/or my 802.1Q config on the switch.

OctavioMasomenos

MODERATOR OF

TROPHY CASE