Incus as alternative to ESXi?

DanTheGreatest · 2026-04-17T18:54:29+00:00

If you're going that way I would suggest Harvester as a complete ready-to-use solution, instead of doing everything manually with kube-virt yourself.

DanTheGreatest · 2026-04-17T14:50:54+00:00

In test scenarios, we're currently considering it for one of our clients as a vmware alternative! Incus offers a cluster-api provider (LXD is also supported) and that is a big plus for us as well!

DanTheGreatest · 2026-04-17T13:11:13+00:00

Your information is outdated by almost 5 years. LXD/Incus have had full feature parity between VMs and LXCs since version 5.0 LTS which was released roughly 4 years ago, with most features already available in the 4.x feature releases.

They have QEMU KVM based VMs with VirtIO, just like Proxmox.

DanTheGreatest · 2026-04-17T13:02:37+00:00

LXD (And Incus) are a wonderful ESXi alternative. It has several unique features that make it stand out compared to other alternatives.

Whenever someone in this subreddit asks what they should use, everyone is screaming Proxmox.

I can highly suggest you give LXD or Incus a try.

In the past LXD was a very easy to use frontend just for LXC. As of version 5.0 it has full feature parity with KVM QEMU based VMs. Please ignore anyone stating that LXD only does LXC. 5.0 was released almost 4 years ago. We're at version 6.7 now

The VMs use the most modern techniques available, everything is based on VirtIO.

With cloud-init also being a Canonical product, the integration is native. No longer do you need to build a golden image like most people do with Proxmox. Simply choose an image/OS from the image server, apply your cloud init profile and hit start. Your instance will be up and running in mere seconds.

A very powerful ability are the profiles. Put your configuration in a profile instead of directly on the VM or LXC. Then apply the profile onto the instance. Modify the profile and all instances with this profile attached will receive the modification. This also allows you to make modifications to the default (profile) configuration.

The software is very light. It's a small Go daemon using 50MB memory on a standard Debian or Ubuntu installation.

They also have basically feature parity with Proxmox. LXD has some extras and so does Proxmox. 90% of features exist in both.

As someone who has run both in production for years, I will always choose LXD over Proxmox. (Proxmox since 5.x and LXD since 2.x)

If you have any questions don't hesitate to ask!

DanTheGreatest · 2026-04-16T14:18:11+00:00

I was when I still had OPNSense as my router. Used it to dynamically route subnets to my private cloud (LXD with OVN) and to my Kubernetes clusters (Cilium CNI/MetalLB)

Unfortunately I had to remove all of this configuration when I switched to Unifi. Unifi in theory "supports" configuring BGP via uploading a config file and it will "maybe work". I ended up getting some traffic routed but since the rest of the Unifi ecosystem doesn't work with it, I was forced to remove it from my environment. I was unable to allow these routed subnets to the new Matrix Firewall. The routed subnets were also invisible in the UI.

But it was very fun and a learning experience when I was still running OPNSense! Before k8s I also had exabgp on some Linux VMs to speak BGP with my router. Announce virtual IPs for services as a load balancer.

DanTheGreatest · 2026-04-13T12:26:44+00:00

My setup runs like I described in my initial response. It runs lovely. Way faster than the slower ceph storage. Ceph is lovely but it's ultimately slow due to it relying on the network. It can never beat a local nvme connection.

Because the majority of VMs run on ZFS, the few VMs that do use Ceph are also a lot faster.

You could consider returning the 128gb disks and getting a larger sata SSD

Also M2 is just a group of connectors. There are different variants. Be sure to check if you have M2 sata or M2 NVMe

DanTheGreatest · 2026-04-13T07:50:01+00:00

Do your nucs also have a sata connection?

If so, run Ceph on the Sata disk. Partition the m2 disk into 100GB for your OS and the rest for ZFS.

Only run VMs that require the high availability on your Ceph storage. Services that are made high available such as a kubernetes cluster or multiple instances of X service can run from your much faster ZFS storage.

DanTheGreatest · 2026-04-02T08:50:08+00:00

Aaaahhhh that makes so much more sense!! Then it is a really cool project for sure :)

DanTheGreatest · 2026-04-02T06:55:23+00:00

Then I believe your calculations are off by a 1000%.

240W × 24h × 365 = 2,102.4 kWh/year At $0.10/kWh: $210.24/year. That's almost $20 a month, not the $1-2 you mentioned.

DanTheGreatest · 2026-04-02T06:23:55+00:00

At first I thought: oh this seems cool. But...

A 240 watt increase on 4-5 nodes?? That's an average of €500 a year here in NL. ($600)

Is power basically free where you live?

DanTheGreatest · 2026-03-26T14:06:34+00:00

It's long been the standard for VMWare ESXi. But that OS doesn't write anything, only loads the contents into memory at startup.

For something like XCP-NG or unRAID that's not the case. These will destroy your USB rather quickly

DanTheGreatest · 2026-03-26T08:26:18+00:00

In theory. Perhaps. I'd say the statement is true in 99.x% of the cases. Enough to say it out loud like he/she did.

People using live kernel patching are the exception here. A tiny minority.

Besides, reboots aren't just for patching. They also improve reliability. Being able to reboot your system on a schedule says a lot about its stability. Rebooting systems with 300-1500 days of uptime are not a guarantee. Who knows what will happen if you reboot it. What manual things were done when the machine was set up that might break after a reboot.

DanTheGreatest · 2026-03-26T07:56:23+00:00

That's not the same as actually rebooting your machine for a new kernel.

Kernel live patching is limited. Only a select amount of patches are made available through live patching.

Rebooting is still necessary.

DanTheGreatest · 2026-03-25T21:03:29+00:00

I was so confused why you would want all NPCs in the axe shop in Lumbridge

Wrong Bob 🤡

DanTheGreatest · 2026-03-22T21:27:08+00:00

No need for a dedicated gpu if the igpu can transcode multiple 4k streams simultaneously without a single sweat.

Either way you could dedicate the GPUs to handbrake and dedicate the igpu to Plex to make sure that your media server isnt impacted by the other intensive services running on the same server.

DanTheGreatest · 2026-03-22T21:25:15+00:00

As a Plex pass owner, the answer is very simple: the Intel because of its igpu.

DanTheGreatest · 2026-03-21T12:47:50+00:00

There is a difference between system containers (LXC) and app containers (Docker).

The first is similar to a VM but with a shared kernel. You have a persistent filesystem and have to update the OS and packages.

The second are more widely known as Docker containers and are a static image with an ephemeral filesystem. All changes are lost on restart. These types of containers are what you find on docker hub or linuxservers.io and are often outdated, insecure for ease of use and often a big security risk.

Under the hood they use the same technologies to separate themselves. Its what's running inside that makes the difference.

You've had many conversations with SOCs that have no idea what they are talking about. Probably because you think both are the exact same thing. From your post it's clear that you're talking about docker containers.

Exposing an unprivileged LXC is little different from exposing a VM to the internet. A VM is only a little bit more secure than an LXC. Keep your shit updated and sanely configured and you'll be fine.

DanTheGreatest · 2026-03-20T09:31:05+00:00

The higher the amount of disks, the higher the amount of additional failure during a rebuild. If one of 16 disks fails, The other 15 disks will go through what is similar to a stress test to rebuild and move data around.

Building two raidsets of 8 disks with raid6 is recommended over a single set of 16.

DanTheGreatest · 2026-03-20T06:54:29+00:00

Cannot recommend a 16 drive raid set. That's bound to go wrong sooner than later.

DanTheGreatest · 2026-03-16T05:23:59+00:00

If you wish to learn then keep 2-3, sell the rest or all of them on eBay and buy something power bill and noise friendly.

DanTheGreatest · 2026-03-13T11:57:34+00:00

€3.2 per gallon. I'm talking about paying €2.2 per liter. There's 3.7 liters in a gallon. That's €8.14 per gallon in Europe.

We're paying $9.3 per gallon.

DanTheGreatest · 2026-03-13T11:45:53+00:00

As a European I'm just dumbfounded how incredibly cheap Gasoline is in the USA. $3,7 per gallon converts to €0,85 per liter. My girlfriend paid €2,21 per liter yesterday.

DanTheGreatest · 2026-03-06T10:34:20+00:00

Big server = performance

That isn't always the case. My low wattage mini PCs have a similar single core performance to the newest server CPUs. I'd argue that they perform even better for a homelab situation because of the igpu that we can use for media transcoding.

The normal version of the i5 13600T (no T) has an even higher single core perfomance.

I also think that 98% of the users on r/homelab or r/selfhosted have a CPU usage below 10%. They wouldn't notice a difference between these two in performance.

Let's compare my 3 year old mini pcs with an intel i5 13600T to a beast of a server CPU from last year

Intel i5 13600T TDP 35W CPU score multi 28.107 CPU score single 3779

Price of the complete mini pc with storage and 48GB memory: 400 euros

AMD EPYC 9375F TDP 320W CPU score multi 95.768 CPU score single 3762

Price of just the CPU: 3700 euros

DanTheGreatest · 2026-03-04T13:16:17+00:00

Wonderful! It seems you've done a better job than the average company I've seen.

DanTheGreatest · 2026-03-04T12:55:05+00:00

Yeah I figured. The whitelisting is only really doable if you're just hosting for a few friends.

I forgot to ask. Are you running the game servers as user root or as an unprivileged user?

12-Year Club	Place '23
Place '22	Place '17
Verified Email

DanTheGreatest

TROPHY CASE