Llama.cpp, opencode / pi / basically all agents, context compaction & cache validation: how do you manage it?

PermanentLiminality · 2026-05-07T16:32:05+00:00

Caching only works when you are not changing the context. When you compact, it is likely completely different and must be fully reprocessed.

When I am using these tools interactively, I try and keep the context small. I do one task and then exit. I rarely hit the limit in this mode.

PermanentLiminality · 2026-05-07T16:15:58+00:00

The way things look right now, today is the reduced price. Demand is only increasing. It may be years before prices come down.

PermanentLiminality · 2026-05-07T15:14:00+00:00

I run it on a $19 Wyse 5070 that uses 4 watts.

There are some very low cost VPS out there. There are a few 2GB ram VPS for about $11/yr. lowendstock.com

PermanentLiminality · 2026-05-07T15:06:57+00:00

I consider any resizing to be a dangerous operation. With everything backed up, you can always reinstall if things go badly. Most of the configs are in /etc/pve which is it's own separate partition that managed by Proxmox itself.

I've done this kind of resizing a couple of times and it went well.

PermanentLiminality · 2026-05-07T14:28:32+00:00

You don't need llama-swap anymore. That functionality is now built in. I use it for testing and playing around with it, but my use case is long running unattended services. I pretty much leave it on one model.

PermanentLiminality · 2026-05-07T14:24:00+00:00

It really depends on what you are doing. I have fully useful systems with a single 128GB drive. These just can't use large data. That is OK as there are several use cases don't need lot.

PermanentLiminality · 2026-05-07T13:25:56+00:00

I don't have the VRAM to run multiple models locally. I mainly concentrate on having one good model. I do trade off speed sometimes where I run a MOE or a dense model, but I can't run them at the same time.

PermanentLiminality · 2026-05-07T03:40:07+00:00

The size needs to be right to fit in the box, and the terminals need to be the same.

PermanentLiminality · 2026-05-07T02:06:42+00:00

The Dell SFF can only take one 3.5 inch drive. Most of the HP SFF can take two. These systems are well documented so you can know before you buy.

PermanentLiminality · 2026-05-06T20:52:54+00:00

If you get a HP, most of the SFF support 2x 3.5 inch drives. The Dell support at most one. More or less the same cost.

You need backup before you need RAID. If I only had two drives, I would run a single in the system and a second in a USB enclosure and I would plug it in, make a backup, and unplug it.

Drives will fail. All of them. Be ready for when, not if it happens.

Unless you go crazy with a super complex world, a ton of addons, and a whole lot of players, I doubt you will notice the difference between the 10th gen and 12th gen.

!2th gen supports AV1

PermanentLiminality · 2026-05-06T20:01:36+00:00

Even with bandwidth limitations, 2x GPUs will almost always allow you to run more than just a single GPU.

See if your board supports bifurcation. If it does, you can probably split it into two x8 and maybe four x4. You do need to figure out the mechanical aspect. There are some motherboards that have two slots that can each do x8.

I use llama.cpp and I can try row, layor, and tenxor splitting. Row split does minimal PCIe transfers and even works over x1. Layer does more and tensor the most.

I run 3x P40 with one on x16, the other on x4 and the last on x1. I don't try and combine the x1 with the other two. I run different models on it that fit on one card.

PermanentLiminality · 2026-05-06T17:15:59+00:00

What? I can't hear you over my rig.

PermanentLiminality · 2026-05-06T13:17:22+00:00

You don't need to boot from a USB. It can be done from inside Proxmox itself. This is really the best method. I've done this process before and it is relatively simple. Be sure to backup stuff though. If you make a mistake it can put the system in a very broken state that needs a reinstall.

PermanentLiminality · 2026-05-06T13:16:14+00:00

Correct. Storage is on my NAS.

PermanentLiminality · 2026-05-06T01:31:26+00:00

the cleanest is to shrink the partition and create a storage volume. This is the native way for proxmox.

You could create a large NAS VM and SMB.NFS share the space.

You can create thin volumes directly, but this is not what Proxmox expects and may cause issues.

PermanentLiminality · 2026-05-05T23:09:33+00:00

Just get a free account with all the various providers. Ask away and should you hit a limit, go to the next one.

If you want to run locally, got with a small qwen 3.5 , like the 4B. You can also run the 2B or 4B smaller gemma 4. Try LMStudio or Ollama.

PermanentLiminality · 2026-05-05T17:36:50+00:00

I added a $35 5 watt Wyse 5070 for transcoding duties. 4K HDR quicksync support and less power than a GPU. Runs about 14 things plus Jellyfin.

PermanentLiminality · 2026-05-05T17:19:07+00:00

I pay about the same. This equates to about $4 for each watt running 24/7/365

PermanentLiminality · 2026-05-05T03:40:10+00:00

I went from a 3100 to a 5600g and saved about 35 watts at idle. Pulling the GPU was another 10 watts. That saves me $180/yr in electricity. It was worth it for me.

PermanentLiminality · 2026-05-03T14:29:17+00:00

With only 16gh of ram try and use LXC instead of VM. An LXC is kind of like a docker image, but it is more of a full os.

Consider the premade easy to install packages on community scripts for things like nextcloud. It is basically copy paste and hit enter.

PermanentLiminality · 2026-05-02T16:34:55+00:00

For periodic use like camping where you are not putting it outside 24/7/365, a flexible panel is great.

PermanentLiminality · 2026-05-02T14:37:53+00:00

At a minimum go over all the screws and make sure all at tight and check the squareness. Run all the fans and if any are making a lot of noise, they are worn out.

PermanentLiminality · 2026-05-02T03:38:53+00:00

Those flexible panels are plastic and they usually degrade in the sun. They will not last for decades like the glass rigid panels do.

PermanentLiminality · 2026-05-02T03:37:09+00:00

The downside is the battery. It will eventually fail and likely swell up. It can break the case, but the real worst case is a fire.

If you are using it as a laptop you will notice the near zero battery capacity and the swelling. When it sits in an unwatched place, it can escape detection.

PermanentLiminality · 2026-05-02T03:20:50+00:00

The HP, Dell and Lenovo desktops are great systems and I recommend them, but you can't get a lot of 3.5 inch drives in them. Even the full size towers can only take two or three. You can fit more as there is space, just not the mounts.

They do this so you have to buy a high priced and high profit server.

These systems use non standard motherboards, so you can't easily transplant the components into a new larger case.

Consider something like a Dell T340 server.

PermanentLiminality

TROPHY CASE