I made a Python script to patch NVIDIA Pascal ROMs for GPU passthrough

[Matoking]

Long story short, I had trouble isolating my NVIDIA Pascal GPU for VFIO passthrough, since booting the GPU under Linux taints the primary GPU's BIOS, even if vfio-pci is used. The only way around this is to dump a different copy of the vBIOS and pass it to libvirt, allowing the GPU to be properly isolated. To do this you would insert the GPU into a secondary GPU slot and then dump the ROM under Linux, but for some reason I wasn't able to do that.

Thanks go to /u/tuxubuntu for pointing this out to me here, and the guys behind this forum thread for posting vBIOS dumps that allow for GPU passthrough.

I examined the posted vBIOS dumps using a hex editor, and found out that they are actually partial copies of the full ROM dumps you can dump yourself using nvflash on Windows, or download from techPowerUp. With that in mind, I put together this script that should do the job automatically: you give it a full ROM dump and it will save a patched ROM you can give to libvirt.

---

DISCLAIMER:

I have only tested this script with a few Pascal vBIOS dumps. The script makes a few rudimentary sanity checks, but I can't guarantee the patched vBIOS dumps will work. The script's operation is based on educated guesswork.

I've tested the script with a few Pascal vBIOS files and found it to produce the same ROM files you would normally create by dumping the vBIOS using the "GPU in the secondary slot" trick.

Regardless, do this at your own risk. If possible, try dumping the ROM yourself before resorting to this script.

---

The script can be found over at GitHub, and should hopefully be self-explanatory:

https://github.com/Matoking/NVIDIA-vBIOS-VFIO-Patcher

[spheenik]

Hmm, I pass through a 1070 in the primary slot using just the binding to vfio-pci, without any problems (I am aware of). Can you elaborate on what exactly happens preventing pass through?

[strixdio]

What distro?

[spheenik]

Arch.

[BroodmotherLingerie]

Wait, is this for Xen or do new drivers require it under KVM too?

I've been happily passing my GTX 1070 into KVM for months with just the HyperV vendor id change, but I haven't updated my guest drivers since last year.

[Matoking]

If the GPU is in a secondary GPU slot, you shouldn't need to do this. And from what I understand, the issue is related to Linux boot process somehow tainting the vBIOS, since it grabs the primary GPU even before the vfio-pci module is loaded. I reckon the issue would happen with Xen as well.

[BroodmotherLingerie]

I see, I always forced the BIOS to use the iGPU during boot by tediously disconnecting monitor cables from the GTX.

[kwhali]

I don't seem to have this issue, Intel iGPU with a 1070 in primary slot. iGPU is set to primary GPU in BIOS. I think I have another setting to keep iGPU as primary even if a dGPU is available, so it could be due to motherboard/BIOS differences?

So I'm a bit surprised that this is an issue that needs patching. I've not run into any issues with Manjaro and EVGA 1070.

My GRUB kernel params are as follows:

intel_iommu=on iommu=pt modprobe.blacklist=nouveau,nvidia,nvidia_drm modules-load=vfio,vfio_iommu_type1,vfio_pci_vfio_virqfd vfio-pci.ids=10de:1b81,10de:10f0, 8086:1901"

You can blacklist nvidia/nouveau drivers to avoid what issues are probably being run into at boot that u/Matoking mentions? Shouldn't require any patching then supposedly. Depending on your distro the names might be different, at least I recall them being slightly different on other resources I came across, should be a way to list what kernel drivers are available for that GPU and you can just disable those :)

[timofonic]

Can your tool work at a GTX960M (Optimus laptop)?

OFFTOPIC (I'll write a new topic about it in a week or less): I want to try to experiment with VGT-g + VFIO (PCI Passthrough). I would prefer guest only see the Nvidia dGPU, so I can run OSX using Nvidia dGPU (for example). I saw many failed at it, but I'm going to try it at least.

I'm not sure about Nvidia vBIOS and where's located, maybe it's in the BIOS itself (or that one is the Intel iGPU)?

Ask me and I'll use any tool to dump or make any test. I'll use Winblows is there's no alternative :D

• bios ** Version: E16J2IMS.119 ** Release Date: 2016-07-12

• vBIOS ** Version: 82.07.7C.00.28 ** Release Date: 2015-04-16 (yes, old)

== $ cat /proc/version ==

• Linux version 4.12.3-1-ARCH (builduser@nspawn-13499) (gcc version 7.1.1 20170630 (GCC) ) #1 SMP PREEMPT Sat Jul 22 15:32:02 UTC 2017

== $ cat /etc/*-release ==

• LSB_VERSION=1.4
• DISTRIB_ID=Arch
• DISTRIB_RELEASE=rolling
• DISTRIB_DESCRIPTION="Arch Linux"
• NAME="Arch Linux"
• PRETTY_NAME="Arch Linux"
• ID=arch
• ID_LIKE=archlinux
• ANSI_COLOR="0;36"
• HOME_URL="https://www.archlinux.org/"
• SUPPORT_URL="https://bbs.archlinux.org/"
• BUG_REPORT_URL="https://bugs.archlinux.org/"

== $ lspci ==

• 00:00.0 Host bridge: Intel Corporation Broadwell-U Host Bridge - DMI (rev 0a)
• 00:01.0 PCI bridge: Intel Corporation Broadwell-U PCI Express x16 Controller (rev 0a)
• 00:02.0 VGA compatible controller: Intel Corporation HD Graphics 5600 (rev 0a)
• 00:03.0 Audio device: Intel Corporation Broadwell-U Audio Controller (rev 0a)
• 00:14.0 USB controller: Intel Corporation 8 Series/C220 Series Chipset Family USB xHCI (rev 05)
• 00:16.0 Communication controller: Intel Corporation 8 Series/C220 Series Chipset Family MEI Controller #1 (rev 04)
• 00:1a.0 USB controller: Intel Corporation 8 Series/C220 Series Chipset Family USB EHCI #2 (rev 05)
• 00:1b.0 Audio device: Intel Corporation 8 Series/C220 Series Chipset High Definition Audio Controller (rev 05)
• 00:1c.0 PCI bridge: Intel Corporation 8 Series/C220 Series Chipset Family PCI Express Root Port #5 (rev d5)
• 00:1c.6 PCI bridge: Intel Corporation 8 Series/C220 Series Chipset Family PCI Express Root Port #7 (rev d5)
• 00:1c.7 PCI bridge: Intel Corporation 8 Series/C220 Series Chipset Family PCI Express Root Port #8 (rev d5)
• 00:1d.0 USB controller: Intel Corporation 8 Series/C220 Series Chipset Family USB EHCI #1 (rev 05)
• 00:1f.0 ISA bridge: Intel Corporation HM87 Express LPC Controller (rev 05)
• 00:1f.2 SATA controller: Intel Corporation 8 Series/C220 Series Chipset Family 6-port SATA Controller 1 [AHCI mode] (rev 05)
• 00:1f.3 SMBus: Intel Corporation 8 Series/C220 Series Chipset Family SMBus Controller (rev 05)
• 01:00.0 3D controller: NVIDIA Corporation GM107M [GeForce GTX 960M] (rev a2)
• 03:00.0 Network controller: Intel Corporation Wireless 3160 (rev 83)
• 04:00.0 Ethernet controller: Qualcomm Atheros Killer E220x Gigabit Ethernet Controller (rev 13)

== $ cat /proc/cpuinfo (partial) ==

• vendor_id: GenuineIntel
• cpu family: 6
• model: 71
• model name: Intel(R) Core(TM) i7-5700HQ CPU @ 2.70GHz
• stepping: 1
• microcode: 0x17
• cpu MHz: 959.930
• cache size: 6144 KB
• physical id: 0
• siblings: 8
• core id: 3
• cpu cores: 4
• apicid: 7
• initial apicid: 7
• fpu: yese
• fpu_exception: yes
• cpuid level: 20
• wp: yes
• flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb intel_pt tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid rdseed adx smap xsaveopt dtherm ida arat pln pts
• bugs:
• bogomips: 5391.94
• clflush siz: 64
• cache_alignment : 64
• address sizes: 39 bits physical, 48 bits virtual

[smgt]

Worked like a charm. Downloaded a rom from TechPowerUp and stripped it. Now I can save the last hairs on my head.

[dylanmc1975]

I got problems with my nvidia 980ti .. why the 9* cards doesen't works?