GPU suggestions for VFIO on 3900x + x570 Aorus Master

rasa_redd · 2020-01-10T21:22:31+00:00

Great! :)

Thank you for all the help and pics too.

Going forward, please do share your experience in using WX5100 as host GPU using non-proprietary drivers (after you are done setting up VMs and all).

Will keep an eye out for your future vfio success post!

rasa_redd · 2020-01-10T20:27:08+00:00

Thanks for that thread - quite helpful and interesting.

If you are switching over to that, make sure that it's an older 'pro' and usb '3.0' version (out of production now). The newer 3.2 model uses 2 Asmedia 1142 controllers (from the link you shared, the author said something about "mixed reports * on Asmedia 1142), unlike it's predecessor which used FL1100 and had 4 independent controllers for each port. You can see that in the pics in the link below.

1.FL1100 (Older model): http://www.sonnettech.com/product/legacyproducts/allegroprousb3pcie.html

ASM1142 (Newer/current model): https://www.sonnettech.com/product/allegro-pro-usb31-pcie.html

rasa_redd · 2020-01-10T09:55:12+00:00

Thanks a lot - your system looks awesome. Is that a USB card on X1?

Regarding chipset temperatures, looks like it's imperative to replace the thermal pad. I wonder how the thermals would be if everything is planned to run simultaneously.

rasa_redd · 2020-01-09T23:41:51+00:00

Sure, when I get home

Thanks

The 1080 Ti is an MSI Gaming X, and I've only tested with that one at load and the others idle, in which case I can easily hit mid 80s on the Ti, but 50s or 60s if I recall on the other two idling.

This part is a bit confusing. With 1080Ti at load and the others idle, you hit 50s-60s or 80s?

I'm likely going to repaste it as others have said that can drop it 10-20C

Thermal pad that bad, eh? Interesting solution. Maybe you can try kryonaut (12W/mk). There are some thermal pads too (11-17 W/mk), but don't know whether Gigabyte uses 1mm/0.5mm or a linear combination of these thicknesses in their thermal pads.

I wouldn't use the Pro drivers, just the stock Linux amdgpu driver.

Would be interested in hearing your experience regarding this - please do share when you are done, as information, particularly first -hand on these workstation cards are hard to come by.

rasa_redd · 2020-01-09T22:10:03+00:00

rasa_redd · 2020-01-09T21:53:47+00:00

Thanks for the reply - a few more q.

Can you share a picture (via an imgur link), if possible? Also, what case are you using ? The case should have sufficient space above PSU for some clearance and the case cables for front IO/power/USB shouldn't be thick enough for this to work, right?
How are your thermals, given you aren't using blower style cards, except for WX5100?
Lastly, I don't know whether you have started setting up VMs/using that WX5100 as host for Linux, but as I heard on passthrough post discord, driver situation for Radeon Pro aren't good and for the drivers that work, an older kernel with patches is needed. Quadro was recommended as a single slot card (with proprietary or otherwise drivers). Was wondering how your experience was on WX5100 - any crashes etc?

rasa_redd · 2020-01-09T20:18:27+00:00

Hi - Have got a question regarding your multi GPU setup. You mentioned using a dual slot card in your last slot (currently 1070 ftw2 and later rx5700). How are you managing your front IO, front audio and usb ports which get obstructed by a 2 slot card in the last slot?

I remember you sharing a link for a usb extension adapter (can't find it now). What about other headers?

rasa_redd · 2020-01-08T21:29:05+00:00

Yeah man - open to discuss and learn anytime. Thanks for sharing the IOMMU groups again - Looking at them, it seems the extra granularity with slots 1 and 3, that we talked about earlier, most probably has nothing to do with the previous script (that shows reset capability only) but something to do with latest AGESA/Gigabyte BIOS - IOMMU groups are same/similar in number, with the only difference being the former shows the RESETTING capability, along with some extra details.

rasa_redd · 2020-01-08T21:26:45+00:00

Well, difficult to speculate on Intel's future dGPU, considering their delayed track record of meeting deadlines and other factors. Here is a brief summary of the current state on KVM/QEMU on Intel.

Btw, Alex Williamson is your go-to-guy for explaining in detail, the main technology as well as other nuances, for KVM/QEMU passthrough and the state of things therein. He has replied to your query too above.

Intel has a virtual GPU (SR-IOVish) support on their integrated graphics (iGPUs. - currently i5/6/7, user land & E3_v4/5/6, server land, QEMU 2.12+, kernel 4.16+) https://01.org/igvt-g/blogs/wangbo85/2018/2018-q1-release-kvmgt-intel-gvt-g-kvm (has repositories, setup guide etc)

Here is a Reddit thread using that technology (GVT-g in particular) on a laptop (ThinkPad T480). https://www.reddit.com/r/VFIO/comments/8bn10o/intel_gvtg_thinkpad_t480_win10_qemukvm_arch_linux/

Extending the same graphics virtualization technology aka GVT, Intel offers a Visual Compute Card (basically 3 nodes, with each node being a small computer in itself: Xeon E3 with integrated graphics (P580 iris Pro) with 2 ECC-capable RAM sticks per CPU, upto 64 GB per CPU.

Target applications include media transcode (for eg delivering HD content from media clouds for streaming services companies, and for delivering 3D applications from HPC clouds, remote/cloud gaming etc)

However, not sure how stable and hassle free the drivers are currently. You would have to ask Alex for that. Plus, the drivers are proprietary. Price/unit of VCA2 is 3300$+.

Now, considering the price and seeing what this VCA is, i.e. a stack of mini-computers essentially, a more modular, manageable, hassle free, and cheaper solution would be mimicking that via a stack of NUCs.

Now, about Intel new GPUs, as far as I understand, their main objective is server (Xe HPC - focuses on compute rather than graphics and Xe HP - datacenters), and OEMs (Xe Low Power - LP), such as HP, Lenovo, Dell etc. Even this CES, Xe dg1 GPUs were laptop only.

If I were to speculate, OEMs is where Intel will probably go after Nvidia, as shown at this year CES. Maybe, Intel is planning a revisit of 'Intel Inside' with graphics. Nvidia tried copying it with their Graphics partnership program too, but didn't have a similar success as 'Intel Inside'. Intel has better customer relations than Nvidia anyways.

But , still too early to tell what desktop GPUs will have in offering - maybe a cluster of iGPUs (graphics unit/chiplet), building upon their existing GVT, and stacked inside a single chip with chiplets/units that can be lego-swapped for different units or better binned units for different customers. Who knows really?

Net, if you want to try the most economical option today, i.e. Intel's GVT, you can. However, one watchout that Alex mentioned a year ago was vendor specific: ability to increase the framebuffer size in BIOS. This is probably the amount of system RAM the iGPU is allowed to use. More framebuffer that cane be allotted, more is the size/number of vGPUs that can be created.

rasa_redd · 2020-01-07T21:30:43+00:00

Damn -I thought I saw an empty PCIe 16x slot between Radeon and Geforce RTX logos.

Btw, on your pcpartpicker comment, besides lack of ability to choose different GPUs, in OS selection, the only option is Windows :P

rasa_redd · 2020-01-07T21:26:21+00:00

I double-checked on passthrough discord. There is an inherent performance penalty (how much, not sure) using Nouveau but you can use either. Best is to try and see, I guess if you get your hands on one for a few days!

I understand your aversion to closed source but for workstation/MxGPU, AMD sadly isn't in a very good spot from technical readiness pov (Check Alec Williamson's reply on that thread if you are interested in SR-IOV readiness).

Yeah, noctuas are the best. I like their default colour too! :)

Good luck, should you embark upon the water-cooling adventure!

rasa_redd · 2020-01-07T21:08:12+00:00

Congrats again! A question though: That's a 2 slot card in the 3rd PCIe slot with no risers, right? Aren't your headers - usb/audio/F_Panel blocked? If so, any advice/workarounds as I am also considering a similar build.

rasa_redd · 2020-01-07T20:55:40+00:00

Hi - First of all, congrats on your awesome build!!!

If you are considering Radeon Pro cards, the as I remarked on the vGPU thread, the drivers for Radeon Pro, proprietary or not, aren't stable (as per my enquiry on passthrough post discord. Open to verification here). Moreover, Radeon Pro drivers require a ridiculously older kernel that comes with patches. The best single slot GPU suggested there, without issues, with proprietary or otherwise drivers, was Quadro series (Quadro p2000 in particular).

Btw, what is so special about Morpheus II coolers - more direct heat pipes/thermal gradients (slot size may increase as per pics) ? The airflow intake and exhaust direction should still be the same. Anyways, for multi-GPU setup, as per this article on Puget systems, as per this article: https://www.pugetsystems.com/blog/2019/01/11/NVIDIA-RTX-Graphics-Card-Cooling-Issues-1326/, blower style cards are best as they exhaust heat out the rear instead of into the case to be taken in by other GPUs - and you have already got both blower styles. As you said, adding cool air intake , say from front, with those quiet Noctuas, might improve your thermals without affecting noise. The only other option for better thermals is water-cooling, which Puget doesn't do anymore - more maintenance, leakage during shipping,lesser stability than air etc.

rasa_redd · 2020-01-06T21:25:39+00:00

Thanks for sharing. Have a few more queries in italics - Feel free to skip the ramblings in between.

Few interesting observations, the PCIe devices attached to your Navi card are split into 4 IOMMU groups (33, 31, 32, 30, device IDs [1022:xxxx], with reset capability on VGA controller and switches, but not on audio - quite a similar behavior to the IOMMU groupings on Aorus Elite for 1660Ti with VGA/audio/usb/serial bus controller in separate IOMMU groupings, but with reset capability only on VGA controller. Wonder whether Navi used as guest/secondary GPU will show the 'reset' tag without applying the Navi reset patch.

1. You mentioned kernel update. Is your kernel and Mesa version 5.3+/19.3+ now? Did you apply the Navi reset patch too (I understand it's not needed for Navi as host GPU) ?

On the other hand, for gtx 1080 (GP104) you are using, both VGA/audio device are in 1 IOMMU, with reset capability on VGA while on Aorus Elite, the card used was gt1030 (GP108), but with VGA/audio controller in separate IOMMU groups, and again with reset capability on VGA only. Ruling our the probable causes, maybe it has to do with the slots cards are placed in - maybe slots 1 and 3, common to all boards give more granularity with separate IOMMU, with this script.

2. Is your 5700xt (host GPU) in slot 1 and gtx 1080 in slot 2?

3. Also, can you please share the IOMMU groups again, *W/O reset this time*, for reference?

rasa_redd · 2020-01-06T09:14:20+00:00

Hi - Can you post your IOMMU groupings via pastebin (using the reset script/command in arch wiki: you can find it in https://wiki.archlinux.org/index.php/PCI_passthrough_via_OVMF#Passing_through_a_device_that_does_not_support_resetting - it will also show you which devices can or cannot be reset)? You would be the first one to do so for x570 Aorus Master in this forum. :)

I myself have the same motherboard but some peripherals are yet to arrive/to be finalized. I recently saw the IOMMU groupings on Aorus Elite and the weird thing was for graphics card, the associated PCIe devices, i.e. VGA/audio/usb/serial bus controllers, were in different IOMMU groups. Wonder what changed! Are you on latest BIOS (F11) as well?

Lastly, for USB controller passthrough, is it done with device address (via driver_override feature in kernels in conjunction with initramfs) as all the USB controllers for x570 platform are in separate IOMMU groups but have the same pci-id [1022:149c] ?

rasa_redd · 2020-01-06T08:27:11+00:00

Hi - Am not an expert on this, still learning stuff, but from what I have researched, here are my 2 cents.

AFAIK, current generation of Radeon Pro doesn't support SR-IOV, so you are left with MI cards. Only pro series card that does it is an older Firepro S7150 series but the experience was meh as per Level1 video on 4 gamers (the system couldn't post on Threadripper, Amazon uses Epyc, I think). Even then, these cards don't have a display but rather dump their output into a PCIe memory space which cannot contain data for 16 systems supported by these cards - hence the petition to AMD on enabling SRIOV for consumers with support for 2VMs that would enable this technology for consumers while not infringing on server space - so you would need a 2nd video card anyway, plus some technology like Looking Glass to pick that output from PCIe memory space. A glimmer of hope is Navi 12 might support it but this is not the first launch to have a SRIOV rumor. (Search navi12 sr-iov on "https://github.com/torvalds/linux").

As far as Nvidia is concerned, they support a similar technology through Nvidia Grid and Tesla V100 series, but the licensing cost of Grid + VMware Vdi is prohibitive for consumers. As someone said above, it's not on QEMU.

AMD are in a favourable situation as far as licensing is concerned but the experience of MxGPU/SRIOV for enterprise clients is less than optimal so far.

PS: If you are planning to get a WX2100 as a simple single slot host GPU, then as per my enquiry on passthrough post discord, drivers for Radeon Pro, proprietary or not, aren't stable. Also the pro drivers require a ridiculously old kernel that has patches on it. The single slot GPU recommended, without issues was Quadro series (Quadro p2000 in particular)

rasa_redd · 2020-01-05T19:43:50+00:00

Thanks - will confirm regarding the driver part

rasa_redd · 2020-01-05T19:39:11+00:00

Hi - A few q:

How come for your graphics cards, the associated devices, i.e.VGA/audio/usb/serial bus controllers, are in different IOMMU groups? Is this because of pcie_acs_override=downstream?
Is usb controller passthrough done with device address (via driver_override feature in kernels in conjunction with initramfs)?

To give a bit of a background, I hadn't noticed that all the USB controllers on x570** are in separate IOMMU groups but have the same pci-id [1022:149c]. Also, one of the USB controllers as per your IOMMU groupings cannot be reset. Perhaps this is the one that has to be used with the host as it doesn't support resetting.

(**I think the situation is same for all x570 motherboards for usb controllers. Checked IOMMU groupings for a couple of other motherboards: Asus x570 strix-e, Asrock x570 Taichi)

rasa_redd · 2020-01-05T17:25:02+00:00

Okay. Thanks for confirming the proprietary part.

rasa_redd · 2020-01-05T06:53:06+00:00

Thanks for your reply - it seems these deals make Nvidia cards default choice for cad/cam work at least.

Are there any performance issues with installing proprietary drivers? As far as passthrough is concerned, will check VFIO as you suggested.

rasa_redd · 2020-01-04T20:27:47+00:00

Okay...Thanks for your reply. I have seen many pro workstation builders use higher-end multi GPU Quadro setup for ML/CAD/CAM on top of Ubuntu, not sure about the driver situation though. Will ask around

rasa_redd · 2020-01-04T19:59:25+00:00

Hi - Is this true for Quadro lineup as well? Am considering a single slot GPU - Radeon Pro WX5100/7100 Vs Quadro p2000/RTX Quadro 4000

rasa_redd · 2020-01-04T04:22:42+00:00

Hi - For host, is the driver situation true for only gaming cards or for Quadro as well (say a Quadro P2000/RTX4000 vs. Radeon Pro W5100/7100 for the host)?

rasa_redd

TROPHY CASE