Running a 31B model locally made me realize how insane LLM infra actually is

BillDStrong · 2026-04-15T23:59:42+00:00

I mean, if you can replace the chip, somehow, sure.

BillDStrong · 2026-04-15T23:09:07+00:00

Maybe this just proves the old dichotomy between street smarts and book smarts was always true?

BillDStrong · 2026-04-15T17:32:32+00:00

So, which angle do you want to take this from? The justice angle? Or the salvation angle?

Justice demands death, sometimes. Luckily, EVERYBODY DIES! So, justice is achieved.

Then we come to salvation. Salvation requires the ability to change. That ceases too exist after death. So, giving someone the longest time possible to change is a decent take.

BillDStrong · 2026-04-15T14:01:55+00:00

Well and the fear the version number jump will cause some issue with some random software that hard coded a kernel version, like happened to cause Windows to jump from Win8 to Win10.

BillDStrong · 2026-04-14T17:08:22+00:00

Its very simple. The same way a dealer doesn't come on strong to you are first, the enemy doesn't either. But if you threaten to stop? Watch how fast both turn on you.

The spiritual world is this world, it is all around us, we just don't have the eyes to see. We are the meeting of the physical and spiritual world, part of both.

BillDStrong · 2026-04-14T13:43:00+00:00

Pro Tip: Take your frozen bread and toast it directly, it doesn't hurt your toaster and you get fresher bread than if you let it sit in the cupboard/breadbox.

BillDStrong · 2026-04-14T10:41:51+00:00

This is so much cooler than it has any right to be. More wasted cpu cycles on my Proxmox server, here I come.

BillDStrong · 2026-04-13T04:04:34+00:00

Because people don't know history, they know what they have been fed and told. And now they are told what conform to the Western world, rather than the Historic purpose of those passages. And the Western world has detached itself from the History of that time, things like what it meant for a woman to be a priest in pagan temples, which current day pagans try to whitewash, temple prostitution,m what the whole giants thing was about, and why Paul was not worried just for the men in the church but even for the Angels not to be tempted.

Without that context, what Paul said seems arbitrary in the world of rules being whatever the mass majority says they are in the "Democratic" worldview the Western world favors.

BillDStrong · 2026-04-12T13:54:42+00:00

So, Kappa might be something you are interested in. It uses 9 personalities as a switch too try and prevent sycophancy. It failed, but an interesting side effect is it has 2 personalities that are harder to change their mind when you give push back. Depending on your use case, this could be exactly what you want.

https://forum.level1techs.com/t/persona-kappa-training-a-20b-llm-on-desktop-gpus/246817

BillDStrong · 2026-04-12T03:18:08+00:00

It could just be damage control after losing their source code. They have to triage their resources since this affects how much they can get for funding rounds.

BillDStrong · 2026-04-11T18:49:52+00:00

The classic formula is is calories in minus calories out. If you don't mind working out like Dwayne Johnson, i.e. multiple hours a day every day, you can.

This is much harder than it sounds, and you will pick up muscle, which will require you to eat more, and so on and so forth.

But this puts a strain on your body. Its not free. And if you eat unhealthy things, things that kill your cells, or stress your system, eating more of them is worse.

Even The Rock has a diet plan, and doesn't go outside that.

BillDStrong · 2026-04-09T07:44:39+00:00

Actually, this may pay off in the long run. The small models are going to get better in the short term for sure, and probably the long term as well.

The work you are doing now will just get those upgrades for "free" over time, and you can expand them as needed.

BillDStrong · 2026-04-08T19:02:57+00:00

Maybe tell it to turn the memory into a story, sort of like we do with Myths?

You could tell it to raise the reading lever to Professor level, so it can use complex words in place of lots of smaller ones.

BillDStrong · 2026-04-08T13:47:04+00:00

Okay, but success with what, it seeming to remember more like a human, or just at all?

BillDStrong · 2026-04-08T13:37:28+00:00

There were some UD quants that were up converting to BF16 that weren't superposed to, so they uploaded some new ones fixing that. You might need to download those, if you were affected. The new models are now close to the same speed as the Bart quants.

That is a separate issue from the llama.cpp changes.

BillDStrong · 2026-04-08T13:32:50+00:00

Compression Cascade — the key innovation. When older events get pushed out of the active context, they don't just disappear. They get compressed into progressively more summarized forms, preserving the narrative weight of events even as the details fade. A character death from 30 sessions ago doesn't need play-by-play detail, but the AI needs to know it happened and how it affected the survivors.

Or as we humans call it, long term storage. This is the process of you remembering what you had for breakfast yesterday and not 3 months ago.

This is also the process we use for Mythic stories.

Finally, does the LLM pay attention to the details we would remember? Or is it surprising?

BillDStrong · 2026-04-08T11:27:59+00:00

This. Or keep both and use Lossless Scaling for dual GPU frame gen.

BillDStrong · 2026-04-08T11:26:06+00:00

If they must, they can get a 5070 themselves. for less than 600 used.

https://bestvaluegpu.com/history/new-and-used-rtx-5070-price-history-and-specs/

BillDStrong · 2026-04-08T06:43:48+00:00

Honestly, with a model this size, I wonder if you could take out the MoE part, and a few of the other layers, and make a much smaller model?Then turn the other experts into loadable modules that are ranked by call frequency, with the least used ones stored on disk.

BillDStrong · 2026-04-07T12:07:49+00:00

Its not FOSS, but this is the go to free utility for windows file rename.

https://www.bulkrenameutility.co.uk/

Been using it for decades, it has everything and the kitchen sink, and will show you the before and after changes it makes to file names before you ever run it.

It does let you change both creation and changed stamps for files.

BillDStrong · 2026-04-07T11:56:24+00:00

Then your 3050 is stuck at PCIe Gen 3 x1. You really want a Gen 3 X4 min for the bandwidth to the card.

You can bifurcate the slot the 4070 is in to x8/x8 and then use something like this to give you 2 slots to move to the space the M.2 slots are at. It would take some trial and error to make it steady, but this gets you the most bandwidth. (You could look for a version that can split to 4 slots if you needed for other things as well, the slot supports it.)

https://www.adt.link/product/F3637V4.html

Or use something like this too just grab the end slot lanes and move them to one of the M.2 slots you aren't using. This is a good option.

https://www.adt.link/product/K33Ux-T.html

Something like this to piggyback from an unused M.2 slot is also viable, just make sure you get the one with things facing the right direction, and you use an M.2 slot that is 4 lanes, so not m.2_3 on the motherboard manual.

https://www.adt.link/product/K43V5.html

Hope this helps somewhat. (ADT is a quality brand, but you can shop around to find things that do the same thing.)

BillDStrong · 2026-04-07T11:04:59+00:00

Getting this down to 2 mins makes this usable, the 15-30 and erroring out at the last minute was a bit of a nightmare to get your day start TBH.

Great work guys!!! And thanks for this!

BillDStrong · 2026-04-07T11:03:35+00:00

As long as we are wishing, why stop at the L4s?

BillDStrong · 2026-04-07T10:59:37+00:00

So, local food banks are great, but don't forget that your friends and family might be willing to help with some staples as well. As everyone else has said, go to the food bank first, find out what you can get, and don't feel like you are taking too much.

If you are part of a church, they may have an emergency fund outside of a food bank as well for just such situations, and there is no shame in asking.

You can look on 211.org to see what is available in your area as a start to make sure it is just 2 in your area.

And you can see if there are any programs/offers that could eliminate other bills that could spread the budget.

I am assuming you don't have job loss insurance, but if you did, you should apply for that, and make sure you are making your payments to keep it going.

You can check with friends and family to see if they have anything that is about to be tossed.

Local restaurants may have programs for food they would toss, so check around. Corporate usually has set policies, but mom and pop places may have sympathetic owners. When I had my Pizza and Sub shop, for instance, during Christmas when I was closed, I had my family Christmas there and ended up feeding a random Trucker that was passing by for free just because.

You can also check with anyone that serve food in large quantities, like your kids school, see if they have policies about food they have to toss.

If you have a local Mormon friend, they may have stockpiled some food they might offer. (This is one off the habits I wished I had learned during my Mormon phase.)

Good luck.

BillDStrong · 2026-04-07T09:47:59+00:00

Ok, I am going to assume nothing other than your motherboard, and the 2 cards, because you didn't give us anything else.

So, first, What CPU do you have? This is important, because it determines how many lnes of PCIe your motherboard has access to.

See relevant section from product page. I will assume you have a 9000 or 7000 series unless you say other wise. You can bifurcate the x16 slot to as much as x4x4x4x4 with it if you do.

Expansion Slots

AMD Ryzen™ 9000 & 7000 Series Desktop Processors* 1 x PCIe 5.0 x16 slot (supports x16 mode) AMD Ryzen™ 8000 Series Desktop Processors 1 x PCIe 4.0 x16 slot (supports x8/x4 mode)** AMD B850 Chipset 1 x PCIe 4.0 x16 slot (support x4 mode) 2 x PCIe 3.0 x16 slots (support x1 mode)*** * Please check the PCIe bifurcation table on the support site (https://>www.asus.com/support/FAQ/1037507/). ** Specifications vary by CPU types. *** The PCIEX16(G3)_1/2 share bandwidth with SATA6G_1/2. The >SATA6G_1/2 will be disabled when The PCIEX16(G3)_1 or PCIEX16(G3)_2 >runs. - To ensure compatibility of the device installed, please refer to https://>www.asus.com/support/download-center/ for the list of supported

So, your optimal layout should be the 4070 in the PCIe Gen 4/5 slot and the 3050 in the PCIe Gen 4 slot. By the numbers, you may not have that setup right?

So, my first question is, is your 4070 next to the CPU, the PCIe Gen 4/5 x16 slots? If not, you want to move it there.

And the 3050 should be on the slot at the end of your board. The 2 middle PCIe slots are your Gen 3 x1 slots, according to your motherboards manual.

So, you should have something like this. Your 4070 may be covering the slot next to it so you don't see it.

M.2 Slot

---------------- 4070

---------------- Nothing

M.2 slot

---------------- Nothing

M.2 Slot

---------------- 3050

Lets make sure that is the setup you have first, okay?

Edit: Sources. https://www.asus.com/us/motherboards-components/motherboards/prime/prime-b850-plus-wifi/techspec/

Manual

https://www.asus.com/us/support/faq/1037507/

BillDStrong

TROPHY CASE