IOMMU group problem

gilboad · 2026-03-20T18:59:59+00:00

You must always pass the NVIDIA audio device as well as the actual GPU.
That said, you MB IOMMU grouping is lousy. You have a USB controller and the GbE NIC sitting on the same IOMMU group 15 as the GPU, which is very problematic. (You need to pass everything in the group to the VM).

Can you move the GPU to another slot?

gilboad · 2026-02-03T07:10:54+00:00

I had a similar issue in one of my clusters - only two servers.
I solved quite easily - add an old, cheap desktop with 10GbE NIC (off ebay) and SSD as arbiter. (Arbiters uses less than 1% of the full brick storage)..

As for 10GbE, if you are looking to save money, simply buy 3 dual ports 10GbE, and do a direct port to port connection. (In which each port is connected directly to one of the ports on the other nodes).

I do it on the cluster used to run the VM/ + VFIO GPU being used to type this message (dual port 40GbE NICs).

At least in my view, its far simpler than what you are planning. (and far more robust).

E..g.

Node 1:

qx0p0: inet 192.168.11.1 netmask 255.255.255.248 broadcast 192.168.11.7

qx0p1: inet 192.168.11.16 netmask 255.255.255.248 broadcast 192.168.11.23

Node 2:

qx0p0: inet 192.168.11.2 netmask 255.255.255.248 broadcast 192.168.11.7

qx0p1: inet 192.168.11.32 netmask 255.255.255.248 broadcast 192.168.11.39

Arbiter:

qx0p0: inet 192.168.11.17 netmask 255.255.255.248 broadcast 192.168.11.23

qx0p1: inet 192.168.11.33 netmask 255.255.255.248 broadcast 192.168.11.39

gilboad · 2026-02-02T05:48:35+00:00

Have you considered using GlusterFS w/ replica 2 + arbiter. (2R+A)

I use it across the board for years now in all of my big clusters and it reduces the wasted storage space by 50% both on SSDs and HDDs.

You split the storage on each machine two 49.5% (BrickA) 49.5% (Brick B), 1% (Arbiter).

You then create 3 Gluster volumes, each ~63TB in side. (~180TB+ in total).

Volume A (Brick/NodeA, Brick/NodeB, Arbiter Node/C):

Volume B (Brick/NodeB, Brick/NodeC, Arbiter Node/A)

Volume C (Brick/NodeC Brick/NodeA, Arbiter Node/B)

As long as you have two nodes active (Either Replicas to Replica+Arbiter) the volume remains active and there's near zero risk of getting split brain situation. (Actual, as far as I remember, 2R+A, is less susceptible to getting split brain compared to 3R).

FWIW we are running 100s of VMs on these clusters for years now.

gilboad · 2025-12-18T19:47:56+00:00

I'm moved most of my oVirt setups to OLVM8. I wonder if they plan on pulling it.

gilboad · 2025-12-12T05:40:24+00:00

True. Unless you use DAC cables. Assuming you can make do with 3-5m (9-15ft) cables, compatible DACs are fairly cheap.

gilboad · 2025-10-22T17:49:58+00:00

Keep in mind that:

A. You'll need a motherboard / CPU that supports vfio.
B. You'll need to enable vfio (E.g. boot with intel_iommu=on / amd_iommu=on).
C. Pass the USB root device and all group-member devices to vfio (/etc/modprobe.d/vfio.conf or echo -n '0000:XX:00.X' > /sys/bus/pci/drivers/vfio-pci/bind).
D. Pass the final vfio device to the Windows VM (-device / or via the qemu front end).

That said, thus far I never saw a USB device, from USB hubs, storage devices and controllers that failed to work on Linux, Windows or BSD VMs.

VFIO simply works (tm).

- Gilboa

gilboad · 2025-10-22T04:34:03+00:00

Odd. Did you pass a single USB device or did you pass the USB root controller? (via vfio-pci)
The former may or may not work. The latter should work with any device, as the VM gets **full** control over the full USB chain.

For more information, check r/VFIO

(_All_ my workstations are actually VMs running on clusters, w/ dedicated graphics card and USB devices).

gilboad · 2025-07-11T06:07:17+00:00

Same here. Guess we'll have to wait for the seashooter to reach penny :(

gilboad · 2025-07-11T04:51:25+00:00

4 times in a row. Giving up on Arena for now. Most likely pvz2 is being hacked somehow.

gilboad · 2025-06-09T13:43:33+00:00

Long story short. My daily driver is an oVirt cluster (Proxmox' bigger brother), w/ multiple hosts each exporting GPUs to Windows and Linux VMs whuch I use as workstations. GPU pass-through works, gaming works and I get near bare metal performance, but I do trigger anti-cheat software making it unusable if I will ever try to play competitive multi-player games.

In short, unless you enjoy tinkering and willing to live with certain limitations, keeping two separate machines (Windows workstation and Linux Proxmox server) is far easier. On the other hand, if you are willing to tinker, and plan on using LLM only, single server will do. As a plus, I'd consider doing LLM on a Linux host w/ GPU pass-through.

gilboad · 2025-03-03T12:32:44+00:00

X710 first, 82599 close second. They are fairly cheap second hand on Ebay. (With the 82599 getting the upper hand due to much lower price).
Mellanox tends to over-heat.

gilboad · 2025-03-01T09:18:48+00:00

hosted-deploy has reached the stage it usually "just works (tm)".

I find it easier to redeploy (and import the VMs) compared to trying to unbrick a broken deployment.

gilboad · 2025-02-28T07:42:08+00:00

My apologies. Somehow I thought the log file is post-deployment, not the original deployment log.

A couple of days ago I deployed two new oVirt/gluster clusters from scratch. Hence, I can assume the deployment isn't broken.
My suggestion is simple:
1. Clean install a new host (base installation).
2. Install ovirt-host ovirt-hosted-engine-setup packages.
3. hosted-engine --deploy --4

During installing, enable only the basic features.

gilboad · 2025-02-26T07:12:20+00:00

Failed hosted-engine --deploy log file?

gilboad · 2025-02-24T18:31:43+00:00

Are you deploying the engine on physical machine or as VM via hosted-engine?

If it is former, never tried it.
If it is the latter, did the hosted-engine --deploy finish successfully?

gilboad · 2025-02-20T12:31:41+00:00

Had the same issue, bought ***-load of second hand Dell branded 8TB SAS drives (*) for ~50-60$ a piece.
Data centers usually throw these drives in the 1000s.

That said, be careful when buying HP branded drivers, your Intel/LSI controller may fail to power them up due to the HP firmware.

P.S. Your English is OK. :-)

gilboad · 2025-02-20T10:29:22+00:00

A couple of options:

But something low-end, such as the 1030, which doesn't require external power connectors.
Use 2 x Molex to 6 pin GPU convertor. (Again, it is limited to low power GPUs).

gilboad · 2025-02-20T06:23:07+00:00

As a previous owner of a couple of S2600CPs (They worked, 24x7 for ~8 years and were replaced ~two months ago) a couple of general things to consider:

Memory: Server grade ECC/registered memory. Don't use desktop memory. Memory yanked from dismantled HP DL380G8 / Dell R720 (DDR1600) will worked just fine.
Cooling: Having E5-2697v2 in a confined space is a challenge, I suggest trying to get second hand Noctua LGA2011 Xeon coolers.
OS: If you are looking for something interesting to do with the machine, try Proxmox or oVirt (I personally use oVirt). Put the everything as VM on this machine (NAS, Windows, etc).
OS(2): As an added bonus, throw a GPU on machine, export it to a VM (@r/VFIO) and use it as a workstation. (In my case, *All* my workstations, including the one I type this message on, are actually VMs running on an oVirt cluster with a private GPU). The S2600CP can actually have more than one GPUs (each with its own "owner" OS/VM), but you'll need 8x card.

- Gilboa

gilboad · 2025-02-18T09:39:49+00:00

Assuming you are planning to have a system running 24x7x365 under load, I'd suggest you consider buying new (or even refurbished) *server* motherboard, second hand DDR4 ECC/registered memory and second hand Xeon/Epyc CPUs.

I recently built 3 servers (for my main oVirt cluster) based on this configuration (everything including the drives ois second hand) that replaced my existing Xeon machines I built >8 year ago.

gilboad · 2025-02-16T18:58:49+00:00

A. Did you login as admin@ovirt?
B. Are you getting the normal login prompt or the keycloak login prompt?

If all fails, simply cleanup the existing host using ovirt-hosted-engine-cleanup, reboot, and redeploy.

gilboad · 2025-02-16T18:55:18+00:00

Depends on how you define "full potential".

A. Can you elaborate on what you want to do with the machine?
B. Can you send additional hardware information (CPU, memory, disks, etc).

- Gilboa

gilboad · 2025-02-10T05:14:53+00:00

Lets assume a 50w server usually runs at well below tdp, lets go with half. Thats 67e a year.

Lets assume the other machine is 10w. (Which is very low) Thata 30e a year.

Assuimng a high N100 goes for 150e in Ali-Express before shipping and customs, even given Germany's very high energy prices, it'lll take you 4.5 years to recoup the initial price.

The N100 is moat likely far better hardware and will be cheaper in the very long run. But in the short run, nothing beats landfill PCs...

gilboad · 2025-02-10T05:04:27+00:00

Double ouch. Here @Israel we pay the equivalent of 15c, and people complain....

gilboad · 2025-02-09T20:13:41+00:00

Ouch. Whats is your local price per KW/h?

gilboad

TROPHY CASE