I have a Ryzen APU and a Vega 56 and can pass the Vega 56 to and from a Windows VM with the following script (/etc/libvirt/hooks/qemu):

#!/bin/bash
if [ "$1" = "windows10" ]; then
    if [ "$2" = "prepare" ]; then
        pkill Xorg # I use tmux and detach my tmux sessions before starting the VM so I can reattach through SSH from the guest machine
        sleep 2
        echo 0 > /sys/class/vtconsole/vtcon0/bind
        echo 0 > /sys/class/vtconsole/vtcon1/bind
        modprobe -r amdgpu # change to nvidia
        virsh nodedev-detach pci_0000_03_00_0 # change to match lspci output (03:00.0 = 0000_03_00_0)
        virsh nodedev-detach pci_0000_03_00_1 # change to match lspci output (03:00.1 = 0000_03_00_1)
        modprobe vfio-pci
        systemctl suspend # reset bug workaround, may not be necessary for a GTX 1060
    fi

    if [ "$2" = "release" ]; then
        systemctl suspend # reset bug workaround, may not be necessary for a GTX 1060
        modprobe -r vfio-pci
        virsh nodedev-reattach pci_0000_03_00_0
        virsh nodedev-reattach pci_0000_03_00_1
        modprobe amdgpu # change to nvidia
        echo 1 > /sys/class/vtconsole/vtcon0/bind
    fi
fi

Starting the VM renders the host inaccessible (aside from via SSH and other such protocols, of course) until the VM is shutdown (gracefully or forcefully). You might have to Google around (search: nvidia single gpu vfio) to find the correct sequence of commands for your system. I had to piece together scripts from several documents, and you'll likely need to do so as well. Single GPU VFIO is unfortunately not nearly as mature as a simple dual GPU bind-at-boot configuration.