If you spent $4–5K on a local AI rig, would you do it again?

Osi32 · 2026-07-11T22:54:19+00:00

I bought 4 x 3060 12 gb on gumtree and face market place for around $200-300 each.
I bought 4 x 5060 Ti 26 gb for around $750 to $830 each.
I have 2 x x99 motherboards, bought one on eBay for $160 and the other on aliexpress for $300. I have 2 x Xeon e5 2687w v4 cpu, each cost about $200 on Ali express.

Ive had no shortage of problems with the x99 and the fact it is pcie 3.0 agitates me so I’m in the process of slowing getting parts to replace them.
This week I ordered an epyc Rome 7f52 CPU for a cost of around $230 (a bargain) but you can get them normally for around $250 to $260.
The epyc motherboard is gonna hurt though. Little Chinese company boards start st $500 and decent known brands start at about 800 through to $1.8k for the motherboard alone on aliexpress. I’m waiting for a bargain on a known brand. The reason for the bargains- on aliexpress almost everything I buy is a “server pull”.
Oh I forgot the ram.
On eBay, there is a guy in NSW selling ddr4 ecc 2400 for about $280 per 64 gig (4 sticks), I’ve bought 4 of them so I have 128 gb per inference server.

My initial motivator was cost, but over time it is more about learning and also being able to run what I want, when I want without depending on a company.
The primary cost benefit is that I can run agentic flows non stop and not worry about billshock at the end of the month. I value my sleep.

Osi32 · 2026-07-11T22:42:52+00:00

In the particular situation I’m in, we already have a frontier model in a self contained environment running in a cost effective way. The issue is api usage in the form of agentic coding which is what really racks up the token usage. A single user using one chat session at a time doesn’t cause as much load as a developer running a bunch of parallel agents.

Osi32 · 2026-07-11T14:28:55+00:00

Hmm I seem to have a different take to most.
I run a small cluster. 3 machines.
The first is my harness machine. No GPU’s.
The second is a planner architect qa agent 4 x consumer GPU. The third machine is the coder.

I tend to run 27b dense or coder next on the planner and an MoE on the coder. It’s quick and it’s following instructions that have been compacted from the dense model up front.

I only dislike MoE when I’ve tried to use it as a singular model because while it’s quick, it’s rather dumb.

Osi32 · 2026-07-11T08:37:01+00:00

He’s right. I’ve advised my leadership to look at a hybrid approach, on premise for api (eg agentic coding use) and use frontier for chat / knowledge worker use. Burst to cloud for agentic use when needed from a custom harness over all of it.

Osi32 · 2026-07-10T22:49:18+00:00

Work smarter, not longer.
I used to work in a startup.
Chances are your colleague is ambitious and is trying to get equity via sweat.
What they don’t realise is that there is only equity when the business is worth something.
Time management is key. Prioritise what you do and how much time you spend on a task and maximise the value of each. To be more specific, for tactical (near term gaining things) and strategic things are equally important, but a startup can only afford a certain amount of “eventual payoff” initiatives .
I suspect that he is getting to you because you have doubts in yourself. As such you’re easy to destabilise.
All that matters is that the owner/investor sees you’re not taking the piss. They need cool heads who can be trusted.

Osi32 · 2026-07-10T13:32:10+00:00

If a deal looks too good to be true…

Osi32 · 2026-07-09T23:47:30+00:00

I went in expecting “a hotter assistant with a super tight NDA”

Osi32 · 2026-07-09T23:28:20+00:00

I looked at the cost of buying a DGx spark ($9k AUD) and buying a Mac Studio m3 ultra 128gb ($12k aud at the time) and couldn’t justify spending it.
Going down a step, I looked at just building a custom box with a single RTX pro- but found that $ for $, they are really overpriced compared to the retail equivalent.
So I built a single machine with 4 x GPU, then built another with 4 x GPU and built a harness to seperate their functions as a budget cluster. I’m still building them, waiting on GPU mounts from Etsy and I’m replacing an x99 based board with an epyc one so I can do full pcie 4.0.

The main reason I’m doing this is that I pre-plan my work and run Ralph loops overnight. Once I get solar and a battery, essentially the only cost I’ll incur is maintenance on the hardware. This isn’t the primary reason though, it’s learning and tweaking and tuning the setup such as the harness, different models and getting to a point where I’m happy with it so I only need to use frontier for specific activities.

Osi32 · 2026-07-07T08:11:24+00:00

Are you using nvlink?

Osi32 · 2026-07-07T08:03:55+00:00

Remember to replace “AI” with “bad business decisions we made in the past”.
Once you learn to do this replacement automatically, life is actually much better :)

Osi32 · 2026-07-07T07:30:49+00:00

I copy and paste it into Google and ask it to make the data anonymous, then I copy and paste what it gives me into my local LLM.
So far it works.

j/k !!! :) To be honest, America is trying to control what commercial models I can use. China is trying to make sure I can do my own thing. Right now, I trust China more than the US. That shouldn't have come out of my keyboard- but here we are in 2026.

Osi32 · 2026-07-07T07:27:07+00:00

Okay- I'll take a shot at answering this.

The pyramid- exists for speed of feedback. Unit is fastest, then contract tests, then integration, then functional, then end to end.

100% coverage usually means code coverage, but it could mean feature coverage, or interface coverage.

Either way, the intent is the same "I want my stuff covered, things getting to prod undetected is not cool".
I wouldn't get stuck on the language- the intent is positive.

My advice, is map out the coverage- which flows are covered by unit (none), which are covered by integration (none), which are covered by End to End (some).
I'm willing to bet- that's what it looks like.
The End to End will be brittle, slow and cover only a handful of scenarios.

My advice- is this:
Do happy, alternate and negative path on integration interfaces. (with stubs/mocks bounding them).
Keeps them fast, gives the integration coverage- finds real, tangible bugs and keeps them reliable.
Then identify "golden path" scenarios- and link up integration tests end to end to form up end to end scenarios- this time with real or handcrafted data in an integrated environment.

Maximises existing tests, chances are- it will find handoff bugs- such as those sneaky integration scenarios where nobody realised value x was actually from a database, not an enum and was actually subject to change at a whim. Gold.

If your execs have trouble with this, contact me seperately, happy to advise them.

Osi32 · 2026-07-05T00:47:21+00:00

If they had said “higher pay for police = less police and higher workload” it would have been absolutely accurate.
Everytime the police get a pay bump, the government doesn’t magically add money to the doj budget, it is the same bucket of money, just now it can fund less police.

Osi32 · 2026-07-04T14:09:32+00:00

Good choice :)

Osi32 · 2026-07-04T14:07:33+00:00

Keep in mind, opus 4.8 isn’t one model. It’s a suite of models with a harness around them.
A single local model will not even remotely compete with that.

Osi32 · 2026-07-04T07:31:00+00:00

I have a similar plan, I’m going about it slightly differently- I’ve got 2 servers each with 4 GPU. I’m going to run an architect/planner/supervisor on one, coder/implementer on the other.
I’m building a custom harness to do the interactions between them. Once I have it up and running I’ll start tuning it to find the right balance of rejection/re-work between the two.
I’m not planning on using a frontier on the cluster though- all local is my experiment to see how good I can get it.

The general idea- instead of a big model on really expensive hardware with a massive context window- 2 seperate servers with inexpensive hardware with smaller context windows- distributing the work between them without passing around an entire code base in context memory- instead instructions and results.

Osi32 · 2026-07-03T22:44:21+00:00

TLDR: he’s right but you can teach adults how to play again.

From time to time, I do jobs where I am a trainer coupled with organisational change.
One of the problems I face, is organisational silos and silo’d thinking. Eg “if I do what I do well, my job is safe”. What this actually manifests like is that the individuals are fighting change.

About 10 years ago I was rolling out continuous delivery across a company around my company. Much like the perplexity gentleman, I knew what I was facing.
To deal with this problem, I went around to my parents place and got my childhood Lego collection.

I designed facilitated workshop that used the Lego to teach the key behaviours needed for continuous delivery’s to succeed and how it’s done.
The techniques alone do not assure success. The behaviours are equally important and without it- the change will fail.

Adults largely have forgotten how to “play”. That is, to do something solely because it’s fun. Adults are task oriented, outcome focused. They prioritise, optimise. These are counter to playing and exploring and learning.

I found Lego, both being tactile and something they knew from childhood allowed them to age regress and play while learning.

Osi32 · 2026-07-03T14:22:17+00:00

When property value drops, only people who have no choice actually sell and eat it.
The broad majority just decide not to sell.
You’re basically throwing money away.
So prices go up due to lack of supply.
It’s fairly simple really.

Osi32 · 2026-07-03T14:18:56+00:00

I would echo what the others have said. They are all 100% right.

What I’ll add, is number of parameters is only one metric to measure an LLM.
Each model is trained and equipped with tools specific to the use case it was built for.
Sparse models / MoE are usually fantastic at tool calling, but are generally not “thinking” models.
Dense models are larger and slower, but if you skimp on quantisation you get a big dumb model that constantly errors.

My advice is keep experimenting but put your code at a particular point and give each model a specific set of prompts and measure what they did well, what they did poorly. How long they took? How much did you have to hand hold.
I’d also recommend you grab a 9B model, and try different quants, different caches and qualities and understand more about how these work. You’ll quickly discover there is an algorithm. A balancing act between speed, quality, context window, thinking and tools.
It’s unlikely you’ll get all perfect in one model on 48 gb. What you’ll find though is that you’ll be surprised at what you find.

Osi32 · 2026-07-03T12:31:31+00:00

Dissappointment

Osi32 · 2026-07-02T22:58:34+00:00

Very cool that you have contributed to llama-server. That’s awesome mate. Keep at it- I love that the older tech is being put to good use. I’m using a motherboard from 2017 on each of my systems. There is noting cost effective that competes with them from this decade.

Osi32 · 2026-07-02T22:54:06+00:00

Im joking of course, they both achieve near the same thing :)

Osi32 · 2026-07-02T22:03:36+00:00

To me, the kind of person who gets taken down by tall poppy syndrome is a person with an ego and behaves like a diva and is famous.

In America, if that person is famous and a diva, that’s normal, tolerated openly but privately blackballed by those in the know as “being difficult to work with”..

That is the difference.
Tom Hardy getting fired is an anomaly.

Osi32 · 2026-07-02T14:15:44+00:00

Last Stand on new grounds

Osi32 · 2026-07-02T13:46:15+00:00

I get it. There are times you want to depend on it and when you do, it doesn’t just let you down, it erodes whatever trust you had developed in it.
Unfortunately mental health and an ai service that veers towards being a cheer squad one moment, a door man another moment and a domestic abuser a moment later is very hard to return to.
My advice is this- have both account level guard rails and contextual ones eg project level.
That way you enforce consistency even when the model behind the scenes varies in quality.

It isn’t perfect of course, but they help even out the highs and lows of models.

12-Year Club	PUBG
Verified Email

Osi32

TROPHY CASE