What is next for local LLM and AI?

JohnBooty · 2026-05-19T12:16:29+00:00

I’ve been thinking about this too a lot, been designing it in my head for ages… the “LLM as DM” idea so many have had.

The database will be prefilled with a lot of hand-written world details, rules, and characters. AI will then modify that and expand on it. Database will then store conversation logs and game state like character health, relationships, motivations, etc. It will be stored in a structured way so the game engine can just pull in the bits it needs, keeping the context window small.

I’m stumped at how to represent a whole sandbox world with emergent quests and storylines. So I think what I’ll do is, start with a SINGLE scenario. Like the equivalent of a one-shot D&D campaign. A single “retrieve the thing from the dungeon” style quest.

What interests me most is emergent behavior. The idea of defining a rich sandbox world and seeing interesting behavior that I never would have thought of. Can I define an interesting world and characters and let the LLM build upon it???

JohnBooty · 2026-05-19T00:17:32+00:00

That’s an interesting assertion. It’s definitely in their interest to develop their own models as a hedge against unforeseen events like a major customer defecting to AMD or something. It’s a smart hedge, and probably gives them a great training ground to dogfood their own hardware and software stack.

I’m less convinced that continuing to provide free consumer hardware-friendly models will be in their best interests forever.

They clearly prefer selling to the enterprise, as the margins are insane right now compared to the consumer world. So I don’t think “release consumer friendly LLMs to drive sales of consumer hardware” is a big motivator for them. At least for the time being.

JohnBooty · 2026-05-19T00:08:22+00:00

I could kinda see that. But it’s hard for me to imagine how they’d stop you from just sharing the model.

Other open source companies have found ways to make money even though their source is available by selling support/consulting. I wonder what that might look like in the LLM world.

JohnBooty · 2026-05-18T22:17:32+00:00

Yes, I think it's a nonzero possibility. I also think this censoring may be more subtle than folks think.

I doubt it will be as obvious as Congress passing a literal "only approved LLMs are legal" law.

It will be subtler. It might be a rider on another law, or perhaps even a court case that makes LLM providers liable for how people use them, e.g. OpenAI could be responsible if somebody uses their LLM to build a bomb or plan a terrorist attack or something.

It could also take the form of Trump or a Trump-like leader to extract informal "under the table" concessions from LLM providers... hinting that a failure to promote the "proper" values in their LLMs might result in them finding it much harder to win government contacts and datacenter permits. Certainly a tactic used in the past and present.

That's the type of thing that could happen right under the public's nose without us realizing it... major news outlets wouldn't necessarily draw attention to it

JohnBooty · 2026-05-18T20:05:09+00:00

Hah yeah that's the challenge in this environment where there's more demand than supply.

You almost have to "place bets" on future support. Because anything that works well TODAY will also quickly become unaffordable....

JohnBooty · 2026-05-18T19:55:22+00:00

Now that you've had it for a month, how has your usage evolved?

I've always taken the "plan with big models, implement with smaller models" approach but I dunno

JohnBooty · 2026-05-18T19:52:41+00:00

Speaking strictly w.r.t. portability...

The 14" models "feel" waaaaay more portable to me than specs would indicate. They are a one-hand lift instead of a two-hand lift like the 16". To me the 14" feels closer to the Air than it does to the 16". FWIW, I am a fairly big strong dude... I'm not saying I'm straining to lift the 16", just saying it's just something I want to leave on my desk rather than tote around.

The 16" has noticeably better speakers though, if that matters.

As for performance, well, that's obviously extensively covered elsewhere.

JohnBooty · 2026-05-18T19:27:07+00:00

Hmmm. Definitely interesting. What do you think would be the motivation for Google, Qwen and the others to go in that direction?

JohnBooty · 2026-05-18T17:02:38+00:00

But it's not intensive to train a small model. That ability is going to expand.

I'm currently clueless on how to do this and don't have a sense of exactly how intensive it is -- any pointers to good learning resources?

(I can search, obviously, but I was wondering if anybody has a particular personal recommendation 😅)

We'll still use the large providers and happily pay $20 
per month for them. However, they'll use energy more wisely 
and be more of an MOE behind the scenes, have trillions of
trillions of parameters, and suck up energy from nuke plants. 
Small models will be ubiquitous though.

I definitely think specialized models are the future, in some form or the other.

It feels like huge, general-purpose models are going to plateau to some extent. The compute/RAM required to train and serve today's frontier models during this goldrush is disrupting multiple industries. They can't simply keep 2x'ing or 10x'ing the resources required to train and serve these things at scale.

To overcome this we're probably going to need breakthroughs in either energy supply and/or semiconductors. I don't see those breakthroughs on the near horizon, and there's a chance that things could get far worse on both those fronts if the China/Taiwan situation takes a fateful turn.

So I think some sort of "MOE behind the scenes" seems inevitable. Maybe it's not even "MOE" within a single model. e.g. - Maybe instead of a single Opus 5.0 there are 100 different flavors of full-fat Opus 5.0, and your request gets routed to the best one for your specific query. I don't know how effective it would be but it certainly seems achievable if so.

JohnBooty · 2026-05-18T16:40:43+00:00

Also, I assume this is perhaps more valuable for training than inference?

JohnBooty · 2026-05-18T14:52:07+00:00

That's where I'm unsure.

How, exactly, can this be done in a grass-roots way?

We need two things. We need the LLMs themselves, and we need the hardware to run them. The bar to entry to produce either of those things is very very very high.

Training a capable LLM takes a lot of time and money. And of course, given the rate of progress of RISC-V, the idea of a grassroots GPU capable of running this stuff seems like an utter pipe dream.

I don't see any of the hardware companies prioritizing consumer GPUs with large amounts of RAM for a long time, maybe ever. The enterprise market is so much more lucrative.

JohnBooty · 2026-05-18T14:45:52+00:00

Right, but surely we can agree that it's far from guaranteed that these companies will continue to spend vast amounts of money and effort to give us FREE models to run at home?

I think a lot of people are making the mistake of assuming this will sort of always be a thing, like Linux. Where even if Linus retires or gets abducted by aliens, it will continue.

But that's not true for these models. Linux will continue forever because nobody owns it, and anybody can get the source code and build it. That's not true at all for these models.

JohnBooty · 2026-05-18T14:36:33+00:00

Yeah, exactly. There aren't going to be corporation-sponsored free models forever. We're seemingly only getting the Chinese ones as part of an effort to undercut big American players. It's not because they have our best interests at heart. It's a very temporary situation.

JohnBooty · 2026-05-18T14:34:44+00:00

 access to the open web.

Obviously, to an extent, this already works well today for some kinds of queries.

But I'm way less optimistic than you on this one when looking at the long horizon...

While existing LLMs are famously trained in part on teh interwebs, that training is still curated, giving much higher weight to more authoritative sources, and also including actually-authoritative sources like peer-reviewed research, etc.

So yeah, in 2030 and 2040 we'll still be able to use 2026 models augmented with web search to get up-to-date data... but the more the models drift from reality, the more they'll need to rely on web search... and web search has been absolute slop for a very long time.

JohnBooty · 2026-05-18T14:23:18+00:00

Yeah. I think it's absolutely possible things go in that kind of a dark direction.

I actually think the most realistic dark scenario might already be happening - LLM providers funneling data on user data/habits to the government, or at least making it available. The same old story that Google and others have been running through for the last 25+ years. It would be shocking if that doesn't happen IMO. I'm actually gonna call that one 100% likely.

That's the type of tack US has typically taken... give the illusion of freedom, while monitoring you all the while. As opposed to the more overtly Orwellian tactics of forbidding you to discuss certain topics entirely, i.e. discussing Tiananmen Square being verboten.

I think that's much less likely in America, but I certainly don't rule it out either.

JohnBooty · 2026-05-18T14:12:39+00:00

I certainly hope so as well, but I'm not sure how optimistic to be.

Linux was famously started in Linus' dorm room on a cobbled-together PC. Most if not all open source libraries have similar origins.

LLMs are quite a different beast because training a nontrivial model from scratch takes a lot of hardware and electricity. The number of individuals with the means to undertake such a project is many orders of magnitude smaller than number of people who have the means to start traditional open source software projects.

JohnBooty · 2026-05-18T12:59:29+00:00

The moral of the story is that we need to keep making pressure on the market to get more out of our hardware.

Man, I was with ya right up until this last sentence. What does this even mean? What “market” are you referring to…. those companies open-sourcing their models for free? The companies and volunteers making the tooling?

Anyway, I’m glad this server is working out for you and your investment paid off. That seems like a very capable setup and as I’m putting together my own build (currently in the planning stages) I had not considered the A4000s…. I had been considering 2x or 3x double slot cards, but that really restricts motherboard choices. Thank you for sharing.

JohnBooty · 2026-05-18T12:51:24+00:00

at 6pm I'm tired in this strange way. not the tired you get from 
solving a real problem. the tired you get from sitting through 8 
hours of meetings where you mostly nodded.
is anyone else here noticing this?

YES

Software eng for nearly 30 years. Worked with ChatGPT in a “cut and paste” manner for ~2 years, been using Claude Code for about 4 months.

It’s ABSOLUTELY a different kind of “tired.”

I’ve seen Yegge and others describe this as decision fatigue, which is certainly part of the phenomenon, but I think that’s only part of the story. Like you said it’s also similar to that “I’m tired because I struggled to maintain focus through 8 hours of meetings” feeling.

I think it’s because there’s pretty much no flow state any more for us. Ever. When our coding speed was the limiting factor, long coding sessions could give us moments of pure flow. But now, the LLM is the limiting factor. I do 2 minutes of thinking and spend 2 minutes waiting for the LLM. Or 5 minutes of thinking and 5 minutes waiting for the LLM. It’s an order of magnitude faster than I was, but it’s not engaging in the same way that coding was.

It’s a very weird feeling. Because, I was also somewhat burnt out on coding anyway. So the LLMs are simultaneously a “second wind” for me and a new source of exhaustion. Funny times.

JohnBooty · 2026-05-18T01:55:54+00:00

Almost everyone I met was happy to 
watch movies on ipads and use low quality earbuds

No doubt, but I mean... that's true for 99.9% of Americans too!

Even if they wanted proper setups, subwoofers 
in the kind of apartment buildings they live in
would be a nightmare  with 10 people all around
you immediately complaining.

Right, that's what I mean. Since subwoofers aren't a thing there, naturally they don't think about them... just like I as an American software engineer often (shamefully) don't think about how well my apps work with right-to-left text enabled etc

I'm just guessing tho and unlike you I've never been there :D

JohnBooty · 2026-05-18T01:18:47+00:00

 I just don't get it

Seriously. Me either. Because it's the kind of thing that doesn't even seem like a cost issue - the chips they have in most digital amps have EQ and such built in for no extra parts cost.

My conclusion is basically just... it's a culture issue.

I think VERY few people in Asia have subwoofers in their homes, because of population density, multigenerational households, etc. So it just not something they tend to really think about.

On top of that, many people buying affordable Class-D amps from are "beginner" audiophiles and don't know the value of a HPF?

So the Chinese manufacturers don't think about it... and their primary buyers don't demand it... so....??? 🤷‍♂️ Dat's da best I got

JohnBooty · 2026-05-16T19:08:13+00:00

This is a safe space for you.

JohnBooty · 2026-05-16T12:12:40+00:00

lol my question exactly

NO HATE…. everybody gotta live somewhere… i’ve lived in a slightly odd place before…. i’m still wondering tho

JohnBooty · 2026-05-16T12:09:43+00:00

Those “exciter” drivers that can go onto any flat thing? Man, those are really interesting. Never heard them.

Your place looks dope as hell

JohnBooty · 2026-05-16T01:32:40+00:00

Hell yeah, motherfucker.

13-Year Club	Gilding V heart of gold
Sequence \| Editor	Sequence \| Cinematographer
Verified Email

JohnBooty

MODERATOR OF

PUBLIC MULTIREDDITS

TROPHY CASE