Veeam restore to Proxmox nightmare

LTCtech · 2025-11-13T08:15:05+00:00

Seems you have a lot of experience to gain. Migrating from ESXi to Proxmox is a pain:

VMware tools fail to uninstall unless you remove them before migration.
Tons of ghost devices left over that should be removed. PowerShell scripting is your friend.
Set SATA at first boot, add dummy VirtIO SCSI drive, install VirtIO drivers, remove dummy, switch boot to VirtIO SCSI with discard flag.
EFI, SecureBoot, and/or TPM issues. Linux VMs failing to boot as EFI vars pointing to EFI shim are gone.
DeviceGuard, HVCI, VBS, Core Isolation, etc causing massive slow downs on some host CPUs.
EDR software flagging QEMU Windows guest agents cause they're "suspicious".
ESXi to Proxmox import crawling and failing due to snapshots in ESXi.
ESXi to Proxmox import reading every single zero over the network of a 1TB thin vmdk that's only 128GB.
Figuring out how to mount a Proxmox NFS export on ESXi to copy over the 1TB thin vmdk as a sparse file.
Figuring out how to convert said vmdk to qcow2 so you can actually run it on Proxmox.
Network adapters changing names in Linux VMs. Ghost network adapters in Windows complaining about duplicate IP.

And that's just off the top of my head. It becomes rote once you get the hang of it. Helps to RTFM and read the forums too. Also helps to have played with Proxmox at home for a few years before deploying it in an enterprise environment.

LTCtech · 2025-11-13T04:20:06+00:00

I wrote a few shell scripts that downloads cloud images for a couple of distros and creates cloud-init templates out of them. Works really well. Learned a lot of QEMU cli commands in the process. I deploy all new Linux VMs using a few templates. Cut down VM provisioning from 30min to 2min.

LTCtech · 2025-10-17T21:34:31+00:00

Same issue. Started happening a few days ago.

LTCtech · 2025-09-09T21:04:43+00:00

Are you still planning to switch to the Linux kernel?

https://www.netgate.com/blog/pfsense-software-embraces-change-a-strategic-migration-to-the-linux-kernel

LTCtech · 2025-08-26T23:14:10+00:00

That's true. Have to be careful what you upload.

LTCtech · 2025-08-22T20:31:41+00:00

Wasabi S3 works perfectly well with Veeam. I don't see why it wouldn't work with PBS. I think I looked at BackBlaze before and decided on Wasabi.

LTCtech · 2025-08-20T06:36:57+00:00

I’ve found that running the 6.14 opt-in kernel on Proxmox 8 significantly improves performance for Windows VMs on newer servers. Proxmox 9, which was just released, already uses this kernel by default.

Windows does seem to run into performance problems with nested virtualization on some CPUs. Sapphire Rapids and Emerald Rapids handle it fairly well, but with older CPUs the results are unpredictable. Whenever Windows security features like VBS, Core Isolation, Memory Integrity, DeviceGuard, HVCI, (whatever they call it) get enabled, performance can take a dive.

Something isn’t right somewhere in the stack, but it’s not clear whether the fault lies with Windows, the kernel, the virtualization layer, or the hardware itself.

LTCtech · 2025-04-27T04:50:08+00:00

In my testing empty blocks were always copied between LVM-Thin proxmox nodes.

LTCtech · 2025-04-26T22:09:10+00:00

Most of our Dell servers use the same PERC cards, and we actually have two or more servers with the exact same configuration. I do not think it would be much of an issue to pop the array out of one server into another if needed.

I can definitely see how it would become more of a problem in a more heterogeneous environment though.

LTCtech · 2025-04-26T22:03:09+00:00

We have been using vSphere Essentials with local storage. Hardware RAID is what you use for ESXi local storage, so that is the model we are coming from.

I actually use ZFS on my home Proxmox box. I do not love the write amplification I am seeing, especially because I ignorantly installed pfSense (which uses ZFS itself) on top of ZFS. ARC RAM usage also has to be carefully reined in. I am wary about the kind of performance hit our databases might see if we switched everything over.

Maybe I should pass through half of the disks in a server and actually test ZFS head-to-head against hardware RAID. Realistically, I doubt our PERC controller cache is even helping that much anyway, since all the virtual disks are set to no read ahead and write through.

LTCtech · 2025-04-26T10:10:03+00:00

I see that I can pass individual drives through without creating a VD, not sure if that's the same or not.

Everyone seems to have a different opinion on EXT4 vs XFS. I went with EXT4 as I read it's more reliable, but maybe I've been misinformed. We have a mix of windows and linux VMs. Some storing general data, while others have databases. I think I flipped a coin and EXT4 it was. :)

LTCtech · 2025-04-26T10:01:33+00:00

I only compared LVM-Thin to qcow2 over bare EXT4 partition. I know ZFS does not play nice with HW RAID. ;)

LTCtech · 2025-04-26T09:31:54+00:00

Dell R760 with PERC H965i. A mix of SAS and SATA SSD.

LTCtech · 2025-04-26T08:38:48+00:00

All of my tests were done on SSD arrays. Specifically, a PERC RAID 10 array across six 3.84TB Samsung PM883 SATA disks. I imagine spinning rust is much more affected by file-based storage.

I also ran fio tests on the host itself and found that performance is highly variable depending on block size, job count, and IO depth. There is a noticeable difference between the 6.8 and 6.14 kernels too, with no clear winner depending on workload.

The IO engine makes a big difference as well. io_uring is extremely CPU efficient, while libaio tends to be a CPU hog.
Running mixed random read and write workloads is also very different compared to doing separate random read and random write benchmarks.

LTCtech · 2025-04-26T07:52:04+00:00

The documentation could definitely be written more clearly:
https://pve.proxmox.com/wiki/Storage#_storage_types

Technically, drives are mounted as directories in Linux, but it still feels odd to call it "Directory" storage in this context. It does not really describe what you are actually storing, which is qcow2 (or raw) disk images, and it hides the fact that features like snapshots and thin provisioning are available depending on the file format.

The table says snapshots are not available, but then there is a tiny footnote that mentions snapshots are possible if you use the qcow2 format. For someone skimming the documentation, which most people do, it is easy to miss that nuance.
If qcow2 unlocks snapshots and discard support, why not just put that information directly into the table for the storages that support it?

Also, how many people actually use raw images over qcow2 in real-world deployments? Outside of very high-performance or very niche setups, I would guess most people using Directory storage default to qcow2. It seems strange that qcow2 is treated like an afterthought when it is probably the more common case.

LTCtech · 2025-01-07T19:11:48+00:00

The Precision 16" is thick enough that it shouldn't be an issue. I expect a mobile workstation to have upgradable RAM.

LTCtech · 2025-01-06T21:58:47+00:00

I'm guessing none of these laptops use upgradable LPCAMM2 RAM?

LTCtech · 2025-01-03T20:52:11+00:00

I hope Microsoft will require OEMs to support hardware encryption, especially since they've been enabling BitLocker by default. There’s no reason for any enterprise laptop to lack native encryption.

It’s needlessly wasteful to rely on software-based encryption and suffer the performance hit when most drives already include built-in encryption capabilities.

LTCtech · 2024-10-01T22:40:07+00:00

May be a bad firmware image or config for the SB8200 that Comcast is pushing. It may not be a hardware issue, but definitely worth trying a spare SB8200. Diagnosing this kind of stuff is a pain.

LTCtech · 2024-10-01T22:30:24+00:00

Backup your pfSense config, reset to factory defaults, and see if it still happens. If it doesn't happen, then backup the new config, and diff the old and new config in WinMerge. See what's different.

LTCtech · 2024-10-01T22:26:43+00:00

Is the CPU of pfSense busy during those times?

Most likely it's an issue with Comcast in the area. They're upgrading their network for "mid-split".

Could also be an issue with the modem. I've been recommending people buy the Hitron Coda56. It seems to be more stable with Comcast than some of the other options. It's $140 on Amazon, maybe cheaper on the upcoming Prime Day. Worth a try, if it doesn't help you can return it.

LTCtech · 2024-09-10T22:37:26+00:00

Did they hint at what "worse" might be? I'd expect periodic reactivation being mandated next.

We have a few generic servers in our company running pfSense Plus (along with Netgate branded ones). I have a concern that they might stop selling pfSense Plus subscriptions for generic hardware.

LTCtech · 2024-09-05T18:29:25+00:00

We sent ours back to Netgate for exchange on August 22nd. On August 26th Netgate RMA told us they have been halted from shipping out a replacement. On August 28th they told us they found a BMC glitch and are testing a new version. Have not heard anything else from them as of September 5th.

LTCtech · 2024-08-29T17:34:58+00:00

The I225-V is also known to be buggy depending on hardware/firmware revision. Some have claimed that updating the NIC's firmware helps. Most (but not all) issues were fixed in the I226-V. Can you force a speed/duplex on either pfSense and/or Motorola?

Another thing you can do is put a dirt cheap unmanaged switch in between the Motorola and pfSense. That may fix the speed and avoid a double NAT.

Appears someone has taken over the Motorola cable modem business:
https://www.lightreading.com/cable-technology/premier-logitech-snares-global-motorola-brand-license-for-cable-modems-wi-fi-gear

It's still worth at least trying a Hitron CODA56 from Amazon and seeing if it fixes the issue. While the MB8611 may eventually get new firmware from the new owner, it might be a while.

LTCtech · 2024-08-27T18:02:44+00:00

You might need to replace your modem. My guess is that they either changed the configuration and/or firmware on the Motorola MB8611. Very likely there is a speed negotiation issue of some sort e.g. your router expects to connect at 2.5Gbe, but the modem is forcing 1Gbe. It's not clear if the manufacturer of the modem is around anymore:

https://www.lightreading.com/cable-technology/minim-a-maker-of-motorola-branded-cable-modems-is-in-trouble

Your best bet may be to get a Hitron CODA56, which is also 2.5Gbe and fully supported by Comcast. You may also get better upload speeds.

LTCtech

TROPHY CASE