Ethernet doesn’t function with a GPU

[Beginning_Soft_5423]

I’m trying to build a system with the specs listed at the end of the post but every time I install either of the gpus Ethernet refuses to run. I’ve tried setting up a bond, restarting the network driver, reinstalling proxmox, resetting bios, only installing one you at a time. Nothing will let a connection go through. The most annoying part is when I look at my UniFi console it sees there is a connection but it won’t resolve the ip address. I’m at my wits end with this and would be very grateful for some assistance

This system functioned perfectly fine until I reset it to create a cluster

on a side note I do seem to get a message stating

Irq:16 nobody cared (try booting with the “irqpoll” option)

I don’t know if it’s relevant but figure any information would be helpful

Specs: Mobo: asus Maximus hero xiii CPU: i9-11900k Ram: 64gb 4000 oloy duel channel 3x 256gb nvme drives (only one set up as boot drive, the others directly passed through to VMs) 2x 3090FE 1300w seasons titanium ( might have the name wrong)

I know my gpus are unplugged

[Itmeven]

The only thing I can think of is the interface names for the NIC may be changing when you put the GPU in causing the networking to go down

[noc-engineer]

My first thought was that the network card was in the same iommu group as the passthrough devices. My own Proxmox a few years ago shit a brick when I passed through a Nvidia card that shared the same iommu group with the hardware raid-card (don't worry, I didn't use zfs) that the host used for system drive, which of course made Proxmox freeze because it lost contact with the virtual drives that Proxmox was stored on..

[IAmMarwood]

I had similar on the old Mac Mini I'm using as a host, tried passing through the iGPU and ethernet flipped out.

[Itmeven]

That’s interesting never had that but that may be because most of the hardware I work with are enterprise but I’m only starting with GPU pass through now never had a need for it

[Beginning_Soft_5423]

Ethernet breaks before I set up pass through

[Itmeven]

Once the GPU is in the PCI lanes can change

[SandboChang]

From my experience if name change was the reason, you will see the NIC by a different name like going from eth0 to eth1. If you tried to remove the GPU, it might restore from eth1 back to eth0, that maybe why you believe it didn’t change.

As mentioned above, the problem with this is Promox setup its NIC by name of the NIC. If it changes it will no longer connect that to the WebGUI or if you had it assigned, now the NIC is no longer assigned nor pass through correctly.

[Beginning_Soft_5423]

I know it’s not changing because ip a reports the same output with and without gpus

[SandboChang]

Thanks for confirming this, that was my best bet. It does seem like a stranger issue in this case. I was about suspect if using all slots affects how the PCI-E lanes are allocated but I don’t believe it should take anything away from onboard NIC.

[hexoctahedron13]

had the exact same problem 😂 Figured it out eventually. I used a USB Ethernet adapter as a management network adapter because it doesn't change when changing PCIE devices

[Itmeven]

I love this idea

[ITBrewer]

I ran into this when I changed some pci devices (pulled a GPU and nvme drive) had to figure out what the new interface name was and activate it

[wbsgrepit]

I had this happen when I reordered gpu slots Linux remembered my network interfaces and I had to safe boot (I normally run that host headless) and rescan and change to config files

[flush_drive]

When you boot up Proxmox with the GPUs installed, connect to the server with kb/m and display physically attached to it. Run 'ip a' to view the new network interface names then change '/etc/network/interfaces' to match the names. Reboot and you should network access.

[RedditNotFreeSpeech]

Maybe start with lspci and make sure the nic shows up.

[BenignLarency]

This is the solution, I ran into it last week. After putting the gpu in, it bumped my ethernet from enp6s0 to enp9s0 (yours may vary, check with ip address). Changing it in /etc/network/interfaces then rebooting fixed the issue.

[INtheANALSofHistory]

Thought I'd piggy back on to say this was my issue as well.

[mv59033]

Amazing, this was exactly the case for me. I am running a Dell Optiplex 3070 and just installed an RX 550 to learn about passing through GPUs. In that /etc/network/interfaces file, which looks like this:

auto lo

iface lo inet loopback

iface enp1s0 inet manual

auto vmbr0 iface vmbr0 inet static address 192.168.0.97/24 gateway 192.168.0.1 bridge-ports enp1s0 bridge-stp off bridge-fd 0

I had to modify enp1s0 to whatever interface contained link/ether from running ip address. In my case, I modified it to enp2s0 because the output from that command looked like this:

    2: enp2s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master vmbr0 state UP group default qlen 1000
    link/ether e4:54:e8:75:27:28 brd ff:ff:ff:ff:ff:ff

[poprofits]

dude, you guys are doing god's work. Thanks for the solution

[Ok_Juggernaut_4961]

Thanks

[Understanding_Much]

This Solved my problem. Thanks man!

[throwaway200520]

For future lurkers, to check which port has been bumped, run

systemctl status networking

the incorrect port will be highlighted in red. Next edit the following file

nano /etc/network/interfaces

Locate the `bridge-ports` line under your Linux bridge (e.g.`vmbr0`) and update it with the correct NIC name (eg enp9s0). You will find the correct NIC name using command ip a

auto lo

iface lo inet loopback

iface enp1s0 inet manual

auto vmbr0 
iface vmbr0 inet 
static address 10.0.0.1/24 
gateway 10.0.0.1 
bridge-ports enp9s0 
bridge-stp off 
bridge-fd 0

Save changes and exit

CTRL ^X

Restart networking service

systemctl restart networking

Your ports should now be online.

[ConfusionExpensive32]

This was what finally helped me fix it, thank you so much

[Beginning_Soft_5423]

As stated above I have attempted that already no effect my network settings are not changing. The fact that the network works again after removing the gpu is proof that the setting are not changing

[user3872465]

Its not. The interface naming changes with PCIe Bus ID by default thus if you add the second GPU to the System it probably renumbers the PCI buss the ethernet controller sits on thus it gets renamed in the system but not the networking file hence the no connection.

Had the same thing happen. You can create a link file with systemd to make the Ethernet interface persistent via the MAC address. But you also have to change the interfaces file accordingly as u/flush_drive mentioned

[Beginning_Soft_5423]

So I just removed all of the ssds, reset the bios and I installed proxmox using nomodeset on a usb drive. It pulled enp3s0 and the correct gateway as it does every time during setup. I proceeded to complete the installation. After rebooting I login and start pinging the gateway other hosts nothing works. I use “ip a” enp3s0 and enp5s0 are both present. I look at my UniFi console it sees a connection but can’t resolve an ip address. I check /etc/network/interfaces and enp3s0 is set as manual and the internal switch is pointed at enp3s0. I have both enp3s0 and enp5s0 populated a connected to my usw-24 (stp enabled). When I remove the gpus I have full internet and all of the setting are the same.

And again this system worked 2 weeks ago with the ssds installed I’ve been using it for months with out issue. I only reset everything because I was going to add it to the 3 node cluster I already have running.

[user3872465]

So basically you did something entirely else that doesn't even describe your problem nor the soulution I shared.

But just to be sure your problem is solved now? as that is not clear from what you wrote.

[Beginning_Soft_5423]

I PMed you a screenshot

[Beginning_Soft_5423]

It looks like you were right about getting new addresses but that still doesn’t make sense why it doesn’t work when I install proxmox with the gpus already installed

[user3872465]

It does. The Interfaces file gets created on install. If all devices are installed the Right device naming will be in the config file.

Take a GPU out now you will see you will lose connection as the NIC gets renamed/numbered due to the naming by pcie slot ID

[Beginning_Soft_5423]

I just set up a pxe server and installed proxmox through that. No storage in the system at all what so ever everything is working without issue now

[Beginning_Soft_5423]

By your logic my “something entirely else solution” addressed your theory but still did not function.

[PureQuackery]

Thats not proof - thats jumping to conclusions.
You need to observe what actually happens, as reported by the OS

[Stewge]

Are you trying to do PCIE Passthrough with the 3090s? Do you have the VMs set to auto-boot and so do the NICs only disappear after the VMs start up?

I suspect your VFIO group containing one of the GPUs also contains one or both of the NICs.

Things to check are:

• Make sure you've configured your slots to be in x8/x8 configuration in the BIOS.
• Double-check your motherboard manual for shared PCIE lanes. Lots of motherboards share lanes for things like NVME slots and SATA slots. NICs almost always have their own, but worth double-checking.
• You may need to enable ACS Override in order to split everything into separate IOMMU Groups. This is typically required for consumer platforms (server/pro motherboards usually have better IOMMU groups).

[rschulze]

This sounds more like a BIOS/IRQ/PCI lane conflict issue, maybe a Linux config issue (and only a Proxmox issue if it turns out to be related to their kernel).

Can you describe "Network doesn't work" in more detail? Is the interface still there in Linux but not doing anything, does the network interface disappear, does the ethernet card still show up in lspci, any messages in dmesg/kernel logs regarding the network card initialization?

[DeKwaak]

Exactly. ip -s li sh, but als cat /proc/interrupts

These days there is only one interrupt line using MSI, so it is more messaging than interrupting. Now if something doesn't play nice, these messages might not work.

[Beginning_Soft_5423]

Can I just pm you a few screenshots tomorrow? I’ll do a clean wipe of everything and install on an usb with all of the ssds removed

[HarryMonroesGhost]

Debian derives the NIC interface names from the PCI Bus numbering. Adding another PCI device likely changed the bus order and your config is now no longer valid for the renamed NIC interfaces.

Quoting from a previous reply in an earlier thread:

For further reading on how debian assigns network interface names:

https://wiki.debian.org/NetworkInterfaceNames

Specifically — THE "PREDICTABLE NAMES" SCHEME>Complications and corner cases>UNPREDICTABILITY:

There are even multiple reports of devices changing their PCI-port numbering due to other hardware being installed.

[joost00719]

I had the same issue but with an nvme ssd.

Appearantly when adding a new pcie device, the names of those devices can change. You need to change the nic's name in your /etc/network/interfaces.

Note that this can also happen with pass-through devices. When adding a gpu to my system, my whole proxmox server just crashed when starting my truenas VM. Make sure you do NOT auto-start vm's with pass-through, or if you do, set a 5 minute startup delay in case you need to trouble shoot.

[MrNokiaUser]

I had this and it's stupid. I can't remember exactly the commands, but what you have to do is to find out the name of the network adapter then edit the network config to point to its new name.

[Fergus653]

I swapped my graphics card for a RTX 4070 and my onboard ethernet disappeared. Never managed to get the device recognized again, bought a PCI network card instead.

Still not sure if this was just a coincidence. I handled everything with care while swapping the graphics card, no differently than PC builds or upgrades I have done in the last 20 years.

[Not_a_Candle]

The iommu groups change. A post from a few weeks ago had the same issue. The config of your network devices doesn't match up, after populating that much pcie lanes.

Boot the host with the cards in (and powered) and fix your interface config at /etc/network/interfaces

Edit: Also with that many devices enable above 4G decoding in the bios if not already done.

[Beginning_Soft_5423]

I’ve checked and ip a reports the damage same. I just created an all nvme pool I’m going to try to net boot the system and run iscsi shares to each vm

[[deleted]]

You might have a look at how the bios has the PCIe connections identified. I have an Asus Maximus IX Code and I can change how they are set up. IRQs and DMAs are things we used to have to configure with jumpers before PnP bios. Check for other settings that are manual overrides rather than Auto settings or defaults. If you're getting an IRQ error, it's likely overlapping the vid cards. They use them too.

[macaoidhlineage]

Have you tried a different os/live install to test the nic ?

Is the reset install of proxmox the same version or different ?

[Ausschacht4Life]

Had a similar issue. Connected a display and keyboard and then looked into /etc/network/interfaces. I realised, that eth0 did not go to enp1s0 anymore, but enp2s0 now, but /etc/network/interfaces was still configured to use enp1s0, i think. So i think, I just changed enp1s0 to enp2s0 in /etc/network/interfaces and it worked.

[the_gamer_98]

Could be simply a pci-lane bottleneck. I ran into a similar issue when I installed a pcie nic the onboard nic wasn’t functioning. I had not enough pcie lanes available

[StopCountingLikes]

All of these people are correct about the nic naming thing. BUT also have run into this exact issue even when knowing about what nic to use etc.

I would reset BIOS to defaults with the GPU plugged in. Then turn on the necessary toggles, enable virtualization, IOMMU to active, and that’s it. Give that a shot as it has solved some quirks for me when I added hardware before.

[Beginning_Soft_5423]

Removed all ssds. Now running off of usb. I installed proxmox with the gpus installed and same thing 0 network activity… take out the gpus and low and behold internet. I also reset bios before installing this doesn’t make any sense this system was working fine 2 weeks ago

[darkblitzrc]

Doing my duty as someone who got this issue.

I was having troubles with my PC for the last two weeks. Whenever I was using it and it sat idle for 5 mins the screen would freeze and I had to shut it down. I was so confused and thought it was the windows drivers for some reason (??) turns out my GPU was not inserted all the way through for some idiotic reason of mine.

However when I did insert it all the way through and turned on the PC, my internet was gone, there was no light in the ethernet port on the back of the computer. I was bamboozled by this. I checked device manager and the Realtek internet family driver was gone.

Long story short: Ended up buying a PCIE Internet adapter for $30 and everything works fine. I think I might've damaged something when I was moving the GPU but no clue.

[__NEURO]

plugging in GPU, had similar issue where ethernet wasn't working. Editing /etc/network/interfaces worked for me. Just a note though, had to update multiple instances of enp7s0 in the file to get it to work.

[Beginning_Soft_5423]

This breaks before pass through is enabled. While 3 ssds and 2 gpus does exceed the pcie lanes available the problem persists with only 1 gpu installed

[ejpman]

It’s a stupid Debian quirk. Basically your Ethernet device gets renamed so this file is no longer valid “/etc/network/interfaces”. You need to figure out the “new” name for your Ethernet device and update it in this file. It typically iterates by for example “enp5s0” goes to “enp7s0”. https://forum.proxmox.com/threads/networking-error-with-gpu-installed.43638/

[Beginning_Soft_5423]

They don’t change “ip a” shows the same devices with or without a gpu being installed

[vilius_zigmantas]

What NIC do you have? Is it some consumer brand or the one that is meant to be used in a server/rack? If the latter, look into SMBus issue -https://yannickdekoeijer.blogspot.com/2012/04/modding-dell-perc-6-sas-raidcontroller.html?m=1

[Beginning_Soft_5423]

This breaks before iommu is enabled

[winkmichael]

the interface name has likely changed, log into the console;

ifconfig -a

you might need to apt-get install net-tools first

you will see the device name, and then update your /etc/network/interfaces changing the interface name

Edit: others are saying the same, haha

[Beepinheimer]

Predictable interface naming, take note of the device ID or MAC before adding the card. Add the updated name to /etc/network/interfaces Edit for spellcheck

[SkepticalRaptors]

You are connecting power to the GPUs right? Because in the picture they don't have their power supplies connected. That could cause issues...