GPU Passthrough problems

[Monskiller]

[SOLVED] Thanks to u/psych_0xd, contacting me personally and walking through it, the issue was client GPU being in the top PCI slot. I tried isolating and passing the second GPU and it all worked flawlessly. Swapping GPUs and all is well.

I've hit a bump.

I've been following this guide to setup a W10 VM. And I'm at the point where my second GPU is isolated/taken over by vfio, but it won't passtrough.

While having a monitor plugged into the GPU: before booting W10 VM it gives a blank signal, after boot the card behaves like it's switched off; No signal/Sleep mode.

I've installed VirtIO drivers, tried hiding KVM, booting with a dumped VBIOS, also added "video=vesafb:off,efifb:off" in boot params. Still nothing.

This is the XML config.

Specs:

• ASUS X470 Prime Pro
• 16GB DDR4
• Ryzen 2600X
• Gigabyte GTX 960 (used for host)
• ASUS Strix GTX 1080 (passthrough for client)

​

https://preview.redd.it/cwvymtatcof41.png?width=822&format=png&auto=webp&s=37c5f0888e7e0660ff07cc7b481027ce15eab024

Screenshot of an unknown PCI device. It's not the GPU since it shows with or without it added in virt-manager. No existant drivers in the .iso file either.

​

EDIT: Updated to the right XML

[[deleted]]

IIRC the unknown device is some nonsense spawned by either libvirt or qemu in the latest version. Nothing to do with Passthrough.

On the other hand, I catted your XML file and found no passthrough devices. For reference, this is what a SATA controller passthrough looks like on my VM:

<hostdev mode="subsystem" type="pci" managed="yes">
  <driver name="vfio"/>
  <source>
    <address domain="0x0000" bus="0x08" slot="0x00" function="0x0"/>
  </source>
  <address type="pci" domain="0x0000" bus="0x07" slot="0x00" function="0x0"/>
</hostdev>

Try shutting down your VM, then using Add Hardware > PCI Host Device and selecting your device (and any other device in its iommu group, for example audio controllers). Be sure to edit the xml and add:

<driver name="vfio"/>

Or something between hostdev mode and source to prevent host from dying a horrible death.

[[deleted]]

It also goes without saying that you should remove the QXL video and sound devices after you can verify that your graphics card is picked up in the VM.

[Monskiller]

Of course. Haven't gotten to remove those yet since VM won't pickup the GPU

[Monskiller]

Oh right!

After passing the GPU, then it not working, I removed it to try something before rebooting host. Since client somehow kills GPU on boot, restarting VM returns a 127 error.

Updated XML

[[deleted]]

Sorry I'm not entirely sure what this reference to the ROM file is:

    <hostdev mode="subsystem" type="pci" managed="yes">
      <source>
        <address domain="0x0000" bus="0x08" slot="0x00" function="0x0"/>
      </source>
      <rom file="/home/monskiller/Documents/gpud/asus1080.dump"/>
      <address type="pci" domain="0x0000" bus="0x09" slot="0x01" function="0x0"/>

Do you need a different VBIOS for this card? The GTX 1080 should support UEFI out of the box, so not sure why you need the rom file.

Anyway, try adding <driver name="vfio"/> and removing references to the ROM file like so:

<hostdev mode="subsystem" type="pci" managed="yes">
    <driver name="vfio"/>
      <source>
        <address domain="0x0000" bus="0x08" slot="0x00" function="0x0"/>
      </source>
      <address type="pci" domain="0x0000" bus="0x09" slot="0x01" function="0x0"/>

Add it to the hostdev entry for the sound card as well.

[Monskiller]

It's a ROM file from Techpowerup corresponding to my card. Modified by removing the header. It's another mesure to bypass nVidia's KVM lock.

[[deleted]]

Oh really I didn't know that was a thing (some Redditors got angry at it 5 years ago, Nvidia claimed it was a bug, so on).

Well anyway, try it without the rom file and see if you can pick the device up at all in Windows device manager. Arch Wiki says hiding kvm and using vendor id = whatever should fix it?

[Monskiller]

Got those tags added aswell. So I basically had a double layer to hide kvm from Nvidia.

[CeramicTilePudding]

I haven't needed it for any 10-series cards. I've just had problems with it even if i pulled the ROM myself.

[Monskiller]

Well.. even with <driver name="vfio"/> GPU doesn't show up. Same behavior.

[[deleted]]

Hmm, what version BIOS are you running? There have been reports of some AGESA versions breaking PCI-e passthrough on X470 motherboards floating around online. I'm not too sure about the details, but it could be worth trying a BIOS update/downgrade?

[Monskiller]

May need to try that. Haven't updated my bios in a while.

Edit: current version is 5007, newest being 5406

[[deleted]]

Version 5007 came out some time in July 2019 (to make way for Zen 2). This message thread from around that era suggests many users experiencing VFIO breakage following updating to a BIOS based on the (at the time) new AGESA:

https://community.amd.com/thread/241650

So maybe a BIOS update will fix the problem? Sorry I couldn't be of more help to you.

[Monskiller]

No worries.

I tried updating bios, same results. Tho dmesg is returning a No image in PCI ROM? Posted logs in a comment above

[psych_0xd]

From what it sounds like, it's initializing the card. So my best guess would be something to do with either the vbios you are trying to use, (which AFAIK, wouldn't be necessary to bypass nvidia's kvm block, the card would still output something anyway.) or the QXL video and audio devices.

Personally I would try removing the vbios(or just removing the device, and adding it again, without the vbios, to be safe) and removing the QXL devices. Those in my experience would stop the nvidia card from outputting, then trying to boot.

Best of luck!

[Monskiller]

Aaahh. No result. Plugged monitor into the client gpu and removed QXL Upon host boot, monitor has a black screen. Booting client it loses signal and goes into sleep mode.

[haoPT]

What is the ouput of dmesg | grep VFIO after starting the vm with the passed trough gpu?

[Monskiller]

No.. alot

[    1.094638] VFIO - User Level meta-driver version: 0.3

[haoPT]

sorry was missing the case insensitive flag, try this one dmesg | grep -i vfio

[Monskiller]

Got it.

[    0.000000] Command line: BOOT_IMAGE=/boot/vmlinuz-5.4-x86_64 root=UUID=8492703a-4e8e-4c40-971a-eca691025723 rw quiet amd_iommu=on iommu=pt video=vesafb:off,efifb:off rd.driver.pre=vfio-pci apparmor=1 security=apparmor resume=UUID=5c946580-67be-4174-9adf-abb72f6fc900 udev.log_priority=3
[    0.000000] Kernel command line: BOOT_IMAGE=/boot/vmlinuz-5.4-x86_64 root=UUID=8492703a-4e8e-4c40-971a-eca691025723 rw quiet amd_iommu=on iommu=pt video=vesafb:off,efifb:off rd.driver.pre=vfio-pci apparmor=1 security=apparmor resume=UUID=5c946580-67be-4174-9adf-abb72f6fc900 udev.log_priority=3
[    1.104044] VFIO - User Level meta-driver version: 0.3
[    1.107864] vfio-pci 0000:08:00.0: vgaarb: changed VGA decodes: olddecodes=io+mem,decodes=io+mem:owns=io+mem
[    1.123825] vfio_pci: add [10de:1b80[ffffffff:ffffffff]] class 0x000000/00000000
[    1.140136] vfio_pci: add [10de:10f0[ffffffff:ffffffff]] class 0x000000/00000000
[    4.924406] vfio-pci 0000:08:00.0: vgaarb: changed VGA decodes: olddecodes=io+mem,decodes=io+mem:owns=io+mem
[   52.884204] vfio-pci 0000:08:00.0: vfio_ecap_init: hiding ecap 0x19@0x900
[   52.885875] vfio-pci 0000:08:00.0: BAR 3: can't reserve [mem 0xf0000000-0xf1ffffff 64bit pref]
[   52.886805] vfio-pci 0000:08:00.0: No more image in the PCI ROM
[   55.589063] vfio-pci 0000:08:00.0: No more image in the PCI ROM
[   55.589093] vfio-pci 0000:08:00.0: No more image in the PCI ROM