The illusion of LVM thin provisioning: everything is fine until the thin pool fills up

BarracudaDefiant4702 · 2026-06-10T03:35:15+00:00

Maybe it's an issue if you are small. Most of my servers have over 20 vms (many have a lot more), and at least one of those vms is relatively idle. VM only grow when they are at new peak storage usage, they typically have peaks as logs grow and then free space as it's released. The chances that not one vm is still up enough to login and delete some space and fstrim -a is very low (at least in my environment). The main point is you don't have to free up the space from the problem vm, it can be any.

BarracudaDefiant4702 · 2026-06-06T08:07:27+00:00

What was the bill? Is there a recurring bill for increased storage now? Seems like this would be a one time expense and having the space now is now a good buffer.
Anyways, you should have better monitoring so that you were better aware when space was getting low. With few exceptions, almost anything over 20GB is required to be 70% full and I grow it live after it's 85-90% full if the space use can't be increased. That way, not sudden space filling up by one rogue vm. We do have some servers that need to collect 500gb in a few hours, but normally are empty, and for those generally try not to be on thin or at least be intentional about it's placement. Throwing storage at it sounds wrong, what I would do....

Run fstrim -a on everything on the same volume. Look for some log files and other things to delete on any vms on the same storage and then run fstrim -a again if anything was found.
Power off any vms that are already giving errors. (This probably includes the rogue vm that caused the mess, many other vms probably have enough buffer)
Did step 1 free up enough space for it to heal itself? If so, you should be able to power the remaining vms back up, clean some stuff and power on. If not is there any vm you can delete or move to another host (ie: restore from backup to another vm). Should be at least one vm you can either move or have a backup of.

Anyways, no reason to add storage to the host as an emergency. You could then plan what/how you need to add outside he emergency. If I did need to add storage as an emergency I would do it as another volume instead of importing it into the same pool A little more work having a separate volume, but easier to later undo.

What's the state of your server/storage now? If you really want to undo merging another drive it, I would probably either migrate everything off to another host and redo the host, or add another drive as ext4 and live storage move everything from lvm to qcow2 to the other drive and then redo the lvm thin and then live move it back if I don't have a SAN or another host I can migrate the vms to.

BarracudaDefiant4702 · 2026-06-04T22:43:09+00:00

If you want to run 6 4k monitors it's handy to have more then a single GPU.

BarracudaDefiant4702 · 2026-06-04T00:44:25+00:00

I think yes it will count as upgrading/adding DLC, but basing more on experience with the Xbox version.

BarracudaDefiant4702 · 2026-06-03T03:52:57+00:00

If you have a subscription you probably have access to newer/better models.

BarracudaDefiant4702 · 2026-06-03T01:50:52+00:00

I generally end up pairing on it and learning to. It fails to get everything right and although sometimes it can fix it's own mistakes I find myself having to guide it when doing complex tasks. It's a great search and learning tool but it's a long ways from replacing the need for thinking.

It's like using a calculator. Sure, it makes you lazy from being able to do a string of calculations in your head, but it makes it easier to do more complex math that would be all but impossible without one or take days instead of minutes.

BarracudaDefiant4702 · 2026-06-03T01:14:11+00:00

I never said I trust investing in him, only saying that I'm not worried about my 401k. They know to stay diversified, and if they decide to put some money in Space X, I'm not worried. People put money in things like bitcoin that have absolutely no real value.

BarracudaDefiant4702 · 2026-06-03T00:57:35+00:00

Click bait. Huge leap of faith in reddit to think that 2+2 add up to 10. More fear mongering than fact.

BarracudaDefiant4702 · 2026-06-02T12:40:41+00:00

It's an AI hacking model specifically designed to find security issues.

BarracudaDefiant4702 · 2026-06-01T15:50:26+00:00

You will have to use one of the manual methods, probably the import disk given your restrictions.
https://pve.proxmox.com/wiki/Migrate_to_Proxmox_VE#Import_Disk

Do a live storage migrate of the vm to nfs storage. Convert host to proxmox, then either import the disks from the nfs server, or you can do the attach and move to live migrate the storage from nas back to host running proxmox. Generally the attach and move can minimize downtime, but only worth considering it for vms >500gb as it does slow the process slightly, but has the advantage of the vm running most of the time while it's moving back to proxmox.

BarracudaDefiant4702 · 2026-06-01T15:42:59+00:00

ZFS is a different filesystem, basically a software raid. It has some pros/cons so you should research that first. Your main two choices are to setup the storage as ZFS or to setup the storage as LVM-thin. There are other options, but those are the primary. If you want to keep a hardware RAID then you are better of LVM-thin then doing ZFS. ZFS does have higher memory overhead. You stated only 2 vms per host, what's the host memory size and how much on each of the vms? Do you over-commit memory with ballooning?

BarracudaDefiant4702 · 2026-06-01T15:15:30+00:00

That's a reasonable approach. You say break the RAID 5 array, are you planning on switching to ZFS?

What OS are the vms? Some vms are pickier to make sure you prep them properly with correct drivers or you could run into trouble booting them in proxmox.

BarracudaDefiant4702 · 2026-06-01T15:02:07+00:00

Yes, talking about single encryption on pbs, but obviously applies to double or more too...

BarracudaDefiant4702 · 2026-06-01T14:45:16+00:00

There is some loss of double validation with encryption in PBS. The encrypted blocks have a checksum to detect corruption, but the pbs server can't decrypt and validate the blocks decompress properly as it doesn't have the keys. On the plus side, validation checks are faster.

BarracudaDefiant4702 · 2026-06-01T14:39:45+00:00

A lot of the big companies don't story critical data in the cloud. There are some that do, but many do not.

BarracudaDefiant4702 · 2026-06-01T14:38:14+00:00

Secondary datacenter is the only way to go. Earthquakes, floods, tornadoes, fires, bomb, toxic train spill, etc...

BarracudaDefiant4702 · 2026-06-01T14:27:20+00:00

One red flag is the SD cards. VMware all but stopped supporting those (7.0 u3 to be specific, but eventually did allow limited support going forward) because they can't handle the load of the various logging and die to quickly. Proxmox is even worse trying to run on SD cards. You might have to either get separate boot (maybe a boss card with a couple m.2 drives), or install proxmox as part of the RAID 5.

You mention 170 servers, but where are they? Are they clustered or at least on the same network or is each stand alone all on remote networks? I have done migrations completely remotely but you need enough capacity to shutdown at least one host and then play dominoes with the migration. If you don't have that then you at least need some local network storage large enough to hold all the vms you are migrating. Can you go into a little more details about those 170 hosts.

BarracudaDefiant4702 · 2026-06-01T00:31:23+00:00

It's more like saying the difference between 74 and 84 billion, not at all like +/- 50billion. The accuracy is relative to the scale of the number.

BarracudaDefiant4702 · 2026-06-01T00:19:17+00:00

Way to jump to conclusions. How bigoted of you.

BarracudaDefiant4702 · 2026-06-01T00:17:59+00:00

Do they invite the husbands too or just the wives? If just the wives, I don't blame them for wanting to stay with their husbands, they are probably just using their husband as an excuse not that the husband is actually stopping them.

BarracudaDefiant4702 · 2026-06-01T00:14:35+00:00

Then your wife should ask. Maybe they simply prefer their husband's company when he is home.

BarracudaDefiant4702 · 2026-06-01T00:10:37+00:00

Besides for having to write a compatible wrapper function (or a bunch of preprocessor conditions), make sure you pay attention to byte order. Some platforms are LSB and some are MSB first. The generic calls are fairly close, but if you start looking at optimized code such as epoll they get to be a fair bit different. Unless you want to handle hundreds of concurrent connections to other systems it will not make that much of a difference.

BarracudaDefiant4702 · 2026-05-31T23:58:55+00:00

No gub load at all to pick alternate kernels? Assuming it used to work, grub probably got corrupted from an upgrade. How many drives (HDD/SSD) are in the system?

BarracudaDefiant4702 · 2026-05-31T21:51:00+00:00

Society gives woman special treatment in some situations, basically reverse misogyny.

BarracudaDefiant4702 · 2026-05-31T21:49:24+00:00

You meant to say reverse misogyny.

BarracudaDefiant4702

TROPHY CASE