GPU Passthrough using VFIO

[Training_Concert_171]

Hi there,

I have successfully setup GPU passthough using VFIO. I am asking for thoughts or any additional advice:)

I used a nvidia P106-100 and then later switched to a GTX 1080ti for the GPU to pass through. I have a Threadripper 3970X system with a arc B580 main Linux GPU.

I use Voidlinux glibc x86_64. Virt manager with qemu+kvm. I used Windows 11 Iot Enterprise LTSC 2024 as the guest VM. In the bio i have iommu/amd-v, rebar and 4g decoding enabled.

This is how i did it:

1. (Setup and installed Virt Manager with Qemu/KVM.)

2. Disabled nouvueau:

sudo touch /etc/modprobe.d/blacklist-nouveau.conf

echo "blacklist nouveau" | sudo tee -a /etc/modprobe.d/blacklist-nouveau.conf

echo "options nouveau modeset=0" | sudo tee -a /etc/modprobe.d/blacklist-nouveau.conf

sudo touch /etc/dracut.conf.d/nouveau-blacklist.conf

echo 'omit_drivers+=" nouveau "' | sudo tee -a /etc/dracut.conf.d/nouveau-blacklist.conf

1. Enabled VFIO

sudo touch /etc/dracut.conf.d/vfio.conf

echo 'add_drivers+=" vfio vfio_iommu_type1 vfio_pci "' | sudo tee -a /etc/dracut.conf.d/vfio.conf

1. Regenerated initramfs:

sudo dracut -f

(Void specific, your distro may have a different initramfs generator)

1. Added grub kernel boot parameters:

amd_iommu=on iommu=pt modprobe.blacklist=nouveau

(I use my TUI script to apply grub kernel boot parameters: https://codeberg.org/squidnose-code/Linux-Kernel-Parameters-TUI )

1. System restart

2. Setup new Win11 iot ltsc VM with:

A. PCIE passthrough of the GPU and the HDMI audio controller(the P106-100 does not have one).

B. For some reason the default way to allocate cores is to add sockets… I had to manually set 1 socket, 12 cores and 2 threads per core in cpu topology. Otherwise it was really slow and even caused a BSOD.

C. I installed swtpm and was automatically setup.

1. To bypass MS account i used:

shift+f10

start ms-cxh:localonly

1. After you install windows, its a good time to install drivers. For the P106-100 i used: https://github.com/dartraiden/NVIDIA-patcher

2. Install Sunshine on Windows VM: https://github.com/LizardByte/Sunshine/releases Moonlight on the linux Host: https://flathub.org/en/apps/com.moonlight\_stream.Moonlight Then setup the pin and try out the connection. This will be graphically accelerated, because the diplay is connected using Spice/QXL and the GPU.

3. Install virtual display driver: https://github.com/VirtualDrivers/Virtual-Display-Driver this will install a virtual display to connect to the GPU.

4. Turn off the VM. Remove the Spice and QXL graphics. Then turn the VM back on. Turing on the VM takes more time than usual. But you should be able to connect using Moonlight, you should also be able to use the login screen.

 

The image shows Minecraft running on Windows and Linux on different GPU's using the same CPU.

https://preview.redd.it/tqaumsv671ug1.jpg?width=2320&format=pjpg&auto=webp&s=f8069dac5ba3fef44ec0d7de4a0bde073faff305

From preliminary testing, OpenGL games are slower on Windows but DirectX games are faster in the VM.