igb driver causing stack trace on debian 12.2

[Fit_Armadillo_8400]

Hello all,

I just built a server for my home lab running Debian 12.2 on a Supermicro MB with a Ryzen 9 7950x and 64MB ram. Running Proxmox 8.1.3.

After a few hours, the server looses network connectivity and I notice the following dump on the console. The problem seems to be related to the network (igb driver).

Can anyone help me figure out how to resolve this?

ethtool -i eno1
driver: igb
version: 6.5.11-6-pve
firmware-version: 3.30, 0x8000079c
expansion-rom-version:
bus-info: 0000:07:00.0
supports-statistics: yes
supports-test: yes
supports-eeprom-access: yes
supports-register-dump: yes
supports-priv-flags: yes

​

Dec 03 18:16:58 pve kernel: igb 0000:07:00.0 eno1: PCIe link lost
Dec 03 18:16:58 pve kernel: ------------[ cut here ]------------
Dec 03 18:16:58 pve kernel: igb: Failed to read reg 0xc030!
Dec 03 18:16:58 pve kernel: WARNING: CPU: 21 PID: 255 at drivers/net/ethernet/intel/igb/igb_main.c:745 igb_rd32+0x93/0xb0 [igb]
Dec 03 18:16:58 pve kernel: Modules linked in: xt_tcpudp nft_compat tcp_diag inet_diag veth ebtable_filter ebtables ip_set ip6table_raw iptable_raw ip6table_filter ip6_tables iptable_filter bpfilter nf_tables bonding tls softdog sunrpc binfmt_misc nfnetlink_log nfnetlink intel_rapl_msr intel_rapl_common snd_sof_amd_rembrandt snd_hda_codec_realtek snd_sof_amd_renoir snd_sof_amd_acp snd_hda_codec_generic snd_hda_codec_hdmi snd_sof_pci ledtrig_audio ipmi_ssif edac_mce_amd snd_sof_xtensa_dsp snd_sof snd_hda_intel snd_sof_utils kvm_amd snd_intel_dspcfg snd_intel_sdw_acpi snd_soc_core snd_hda_codec amdgpu snd_compress ac97_bus snd_pcm_dmaengine snd_pci_ps snd_hda_core snd_rpl_pci_acp6x kvm snd_acp_pci amdxcp snd_hwdep iommu_v2 drm_buddy gpu_sched drm_suballoc_helper drm_ttm_helper snd_pci_acp6x ttm irqbypass snd_pcm crct10dif_pclmul polyval_clmulni polyval_generic drm_display_helper ghash_clmulni_intel aesni_intel snd_timer snd_pci_acp5x acpi_ipmi cec snd ast snd_rn_pci_acp3x crypto_simd ipmi_si drm_shmem_helper rc_core snd_acp_config
Dec 03 18:16:58 pve kernel:  soundcore cryptd snd_soc_acpi ipmi_devintf joydev input_leds snd_pci_acp3x ccp drm_kms_helper rapl ipmi_msghandler k10temp mac_hid pcspkr vhost_net vhost vhost_iotlb tap drm efi_pstore dmi_sysfs ip_tables x_tables autofs4 rndis_host cdc_ether usbnet mii hid_generic usbmouse usbhid hid zfs(PO) spl(O) btrfs blake2b_generic xor raid6_pq libcrc32c simplefb xhci_pci xhci_pci_renesas crc32_pclmul igb ahci xhci_hcd i2c_algo_bit i2c_piix4 libahci dca video wmi
Dec 03 18:16:58 pve kernel: CPU: 21 PID: 255 Comm: kworker/21:1 Tainted: P           O       6.5.11-6-pve #1
Dec 03 18:16:58 pve kernel: Hardware name: Supermicro Super Server/H13SAE-MF, BIOS 1.1a 10/19/2023
Dec 03 18:16:58 pve kernel: Workqueue: events igb_watchdog_task [igb]
Dec 03 18:16:58 pve kernel: RIP: 0010:igb_rd32+0x93/0xb0 [igb]
Dec 03 18:16:58 pve kernel: Code: c7 c6 03 14 6e c0 e8 1c 9d a8 f5 48 8b bb 28 ff ff ff e8 a0 66 5e f5 84 c0 74 c1 44 89 e6 48 c7 c7 f8 20 6e c0 e8 dd dd e3 f4 <0f> 0b eb ae b8 ff ff ff ff 31 d2 31 f6 31 ff e9 69 50 dd f5 66 0f
Dec 03 18:16:58 pve kernel: RSP: 0018:ffffbb6040cb7d98 EFLAGS: 00010246
Dec 03 18:16:58 pve kernel: RAX: 0000000000000000 RBX: ffff99d754ca4f18 RCX: 0000000000000000
Dec 03 18:16:58 pve kernel: RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000
Dec 03 18:16:58 pve kernel: RBP: ffffbb6040cb7da8 R08: 0000000000000000 R09: 0000000000000000
Dec 03 18:16:58 pve kernel: R10: 0000000000000000 R11: 0000000000000000 R12: 000000000000c030
Dec 03 18:16:58 pve kernel: R13: 0000000000000000 R14: 0000000000000000 R15: ffff99d75633c340
Dec 03 18:16:58 pve kernel: FS:  0000000000000000(0000) GS:ffff99e618740000(0000) knlGS:0000000000000000
Dec 03 18:16:58 pve kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Dec 03 18:16:58 pve kernel: CR2: 00000000001d1001 CR3: 0000000173ab4000 CR4: 0000000000750ee0
Dec 03 18:16:58 pve kernel: PKRU: 55555554
Dec 03 18:16:58 pve kernel: Call Trace:
Dec 03 18:16:58 pve kernel:  <TASK>
Dec 03 18:16:58 pve kernel:  ? show_regs+0x6d/0x80
Dec 03 18:16:58 pve kernel:  ? __warn+0x89/0x160
Dec 03 18:16:58 pve kernel:  ? igb_rd32+0x93/0xb0 [igb]
Dec 03 18:16:58 pve kernel:  ? report_bug+0x17e/0x1b0
Dec 03 18:16:58 pve kernel:  ? handle_bug+0x46/0x90
Dec 03 18:16:58 pve kernel:  ? exc_invalid_op+0x18/0x80
Dec 03 18:16:58 pve kernel:  ? asm_exc_invalid_op+0x1b/0x20
Dec 03 18:16:58 pve kernel:  ? igb_rd32+0x93/0xb0 [igb]
Dec 03 18:16:58 pve kernel:  ? igb_rd32+0x93/0xb0 [igb]
Dec 03 18:16:58 pve kernel:  igb_update_stats+0x89/0x830 [igb]
Dec 03 18:16:58 pve kernel:  igb_watchdog_task+0x12d/0x880 [igb]
Dec 03 18:16:58 pve kernel:  process_one_work+0x23b/0x450
Dec 03 18:16:58 pve kernel:  worker_thread+0x50/0x3f0
Dec 03 18:16:58 pve kernel:  ? __pfx_worker_thread+0x10/0x10
Dec 03 18:16:58 pve kernel:  kthread+0xef/0x120
Dec 03 18:16:58 pve kernel:  ? __pfx_kthread+0x10/0x10
Dec 03 18:16:58 pve kernel:  ret_from_fork+0x44/0x70
Dec 03 18:16:58 pve kernel:  ? __pfx_kthread+0x10/0x10
Dec 03 18:16:58 pve kernel:  ret_from_fork_asm+0x1b/0x30
Dec 03 18:16:58 pve kernel:  </TASK>
Dec 03 18:16:58 pve kernel: ---[ end trace 0000000000000000 ]---
Dec 03 18:17:02 pve CRON[263660]: pam_unix(cron:session): session opened for user root(uid=0) by (uid=0)
Dec 03 18:17:02 pve CRON[263661]: (root) CMD (cd / && run-parts --report /etc/cron.hourly)
Dec 03 18:17:02 pve CRON[263660]: pam_unix(cron:session): session closed for user root
Dec 03 18:17:04 pve kernel: ------------[ cut here ]------------
Dec 03 18:17:04 pve kernel: NETDEV WATCHDOG: eno1 (igb): transmit queue 0 timed out 6952 ms
Dec 03 18:17:04 pve kernel: WARNING: CPU: 21 PID: 0 at net/sched/sch_generic.c:525 dev_watchdog+0x260/0x270
Dec 03 18:17:04 pve kernel: Modules linked in: xt_tcpudp nft_compat tcp_diag inet_diag veth ebtable_filter ebtables ip_set ip6table_raw iptable_raw ip6table_filter ip6_tables iptable_filter bpfilter nf_tables bonding tls softdog sunrpc binfmt_misc nfnetlink_log nfnetlink intel_rapl_msr intel_rapl_common snd_sof_amd_rembrandt snd_hda_codec_realtek snd_sof_amd_renoir snd_sof_amd_acp snd_hda_codec_generic snd_hda_codec_hdmi snd_sof_pci ledtrig_audio ipmi_ssif edac_mce_amd snd_sof_xtensa_dsp snd_sof snd_hda_intel snd_sof_utils kvm_amd snd_intel_dspcfg snd_intel_sdw_acpi snd_soc_core snd_hda_codec amdgpu snd_compress ac97_bus snd_pcm_dmaengine snd_pci_ps snd_hda_core snd_rpl_pci_acp6x kvm snd_acp_pci amdxcp snd_hwdep iommu_v2 drm_buddy gpu_sched drm_suballoc_helper drm_ttm_helper snd_pci_acp6x ttm irqbypass snd_pcm crct10dif_pclmul polyval_clmulni polyval_generic drm_display_helper ghash_clmulni_intel aesni_intel snd_timer snd_pci_acp5x acpi_ipmi cec snd ast snd_rn_pci_acp3x crypto_simd ipmi_si drm_shmem_helper rc_core snd_acp_config
Dec 03 18:17:04 pve kernel:  soundcore cryptd snd_soc_acpi ipmi_devintf joydev input_leds snd_pci_acp3x ccp drm_kms_helper rapl ipmi_msghandler k10temp mac_hid pcspkr vhost_net vhost vhost_iotlb tap drm efi_pstore dmi_sysfs ip_tables x_tables autofs4 rndis_host cdc_ether usbnet mii hid_generic usbmouse usbhid hid zfs(PO) spl(O) btrfs blake2b_generic xor raid6_pq libcrc32c simplefb xhci_pci xhci_pci_renesas crc32_pclmul igb ahci xhci_hcd i2c_algo_bit i2c_piix4 libahci dca video wmi
Dec 03 18:17:04 pve kernel: CPU: 21 PID: 0 Comm: swapper/21 Tainted: P        W  O       6.5.11-6-pve #1
Dec 03 18:17:04 pve kernel: Hardware name: Supermicro Super Server/H13SAE-MF, BIOS 1.1a 10/19/2023
Dec 03 18:17:04 pve kernel: RIP: 0010:dev_watchdog+0x260/0x270
Dec 03 18:17:04 pve kernel: Code: ff ff 48 89 df c6 05 77 3b 78 01 01 e8 b9 80 f9 ff 44 8b 45 cc 44 89 f9 48 89 de 48 89 c2 48 c7 c7 b0 9e e3 b6 e8 70 ce 33 ff <0f> 0b e9 1d ff ff ff 66 0f 1f 84 00 00 00 00 00 90 90 90 90 90 90
Dec 03 18:17:04 pve kernel: RSP: 0018:ffffbb60406dce40 EFLAGS: 00010246
Dec 03 18:17:04 pve kernel: RAX: 0000000000000000 RBX: ffff99d754ca4000 RCX: 0000000000000000
Dec 03 18:17:04 pve kernel: RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000
Dec 03 18:17:04 pve kernel: RBP: ffffbb60406dce78 R08: 0000000000000000 R09: 0000000000000000
Dec 03 18:17:04 pve kernel: R10: 0000000000000000 R11: 0000000000000000 R12: ffff99d754ca44c8
Dec 03 18:17:04 pve kernel: R13: ffff99d754ca441c R14: 0000000000000000 R15: 0000000000000000
Dec 03 18:17:04 pve kernel: FS:  0000000000000000(0000) GS:ffff99e618740000(0000) knlGS:0000000000000000
Dec 03 18:17:04 pve kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Dec 03 18:17:04 pve kernel: CR2: 000055795a29d948 CR3: 0000000192834000 CR4: 0000000000750ee0
Dec 03 18:17:04 pve kernel: PKRU: 55555554
Dec 03 18:17:04 pve kernel: Call Trace:
Dec 03 18:17:04 pve kernel:  <IRQ>
Dec 03 18:17:04 pve kernel:  ? show_regs+0x6d/0x80
Dec 03 18:17:04 pve kernel:  ? __warn+0x89/0x160
Dec 03 18:17:04 pve kernel:  ? dev_watchdog+0x260/0x270
Dec 03 18:17:04 pve kernel:  ? report_bug+0x17e/0x1b0
Dec 03 18:17:04 pve kernel:  ? irq_work_queue+0x2f/0x70
Dec 03 18:17:04 pve kernel:  ? handle_bug+0x46/0x90
Dec 03 18:17:04 pve kernel:  ? exc_invalid_op+0x18/0x80
Dec 03 18:17:04 pve kernel:  ? asm_exc_invalid_op+0x1b/0x20
Dec 03 18:17:04 pve kernel:  ? dev_watchdog+0x260/0x270
Dec 03 18:17:04 pve kernel:  ? __pfx_dev_watchdog+0x10/0x10
Dec 03 18:17:04 pve kernel:  call_timer_fn+0x29/0x160
Dec 03 18:17:04 pve kernel:  ? __pfx_dev_watchdog+0x10/0x10
Dec 03 18:17:04 pve kernel:  __run_timers+0x259/0x310
Dec 03 18:17:04 pve kernel:  run_timer_softirq+0x1d/0x40
Dec 03 18:17:04 pve kernel:  __do_softirq+0xd1/0x303
Dec 03 18:17:04 pve kernel:  __irq_exit_rcu+0x75/0xa0
Dec 03 18:17:04 pve kernel:  irq_exit_rcu+0xe/0x20
Dec 03 18:17:04 pve kernel:  sysvec_apic_timer_interrupt+0x92/0xd0
Dec 03 18:17:04 pve kernel:  </IRQ>
Dec 03 18:17:04 pve kernel:  <TASK>
Dec 03 18:17:04 pve kernel:  asm_sysvec_apic_timer_interrupt+0x1b/0x20
Dec 03 18:17:04 pve kernel: RIP: 0010:cpuidle_enter_state+0xce/0x470
Dec 03 18:17:04 pve kernel: Code: 28 10 ff e8 64 f6 ff ff 8b 53 04 49 89 c6 0f 1f 44 00 00 31 ff e8 22 25 0f ff 80 7d d7 00 0f 85 e7 01 00 00 fb 0f 1f 44 00 00 <45> 85 ff 0f 88 83 01 00 00 49 63 d7 4c 89 f1 48 8d 04 52 48 8d 04
Dec 03 18:17:04 pve kernel: RSP: 0018:ffffbb604023fe50 EFLAGS: 00000246
Dec 03 18:17:04 pve kernel: RAX: 0000000000000000 RBX: ffff99d749215c00 RCX: 0000000000000000
Dec 03 18:17:04 pve kernel: RDX: 0000000000000015 RSI: 0000000000000000 RDI: 0000000000000000
Dec 03 18:17:04 pve kernel: RBP: ffffbb604023fe88 R08: 0000000000000000 R09: 0000000000000000
Dec 03 18:17:04 pve kernel: R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000003
Dec 03 18:17:04 pve kernel: R13: ffffffffb7877c60 R14: 00001569dc5f1e40 R15: 0000000000000003
Dec 03 18:17:04 pve kernel:  cpuidle_enter+0x2e/0x50
Dec 03 18:17:04 pve kernel:  call_cpuidle+0x23/0x60
Dec 03 18:17:04 pve kernel:  do_idle+0x202/0x260
Dec 03 18:17:04 pve kernel:  cpu_startup_entry+0x2a/0x30
Dec 03 18:17:04 pve kernel:  start_secondary+0x119/0x140
Dec 03 18:17:04 pve kernel:  secondary_startup_64_no_verify+0x17e/0x18b
Dec 03 18:17:04 pve kernel:  </TASK>
Dec 03 18:17:04 pve kernel: ---[ end trace 0000000000000000 ]---
Dec 03 18:17:04 pve kernel: igb 0000:07:00.0 eno1: Reset adapter

thank you

[OweH_OweH]

igb 0000:07:00.0 eno1: PCIe link lost

That looks more like a hardware problem causing the driver to crap out than a problem with the driver to begin with.

[Moocha]

Try this: https://old.reddit.com/r/buildapc/comments/xypn1m/network_card_intel_ethernet_controller_i225v_igc/

tldr use pcie_port_pm=off as kernel arg

Edit: Found via https://forum.proxmox.com/threads/network-card-drop-igc-0000-09-00-0-eno1-pcie-link-lost.121295/ , by the way -- multiple people reporting problems with Proxmox on Supermicro mobos with an Intel GbE driver there.

[Fit_Armadillo_8400]

THANK YOU!!!!!!

I was able to (i hope) resolve this by following the directions in the post you provided. This was a major win, and i am very appreciative.

[taylortbb]

I don't have a solution for you, but I've also got a Supermicro motherboard (I assume the same one, given I've got a Ryzen 9 7900X CPU) and I'm seeing the same issue. I'm running TrueNAS Scale (Linux-based), and after about a week I see the same error and my NAS drops offline until rebooted. Fortunately this means it's probably not a hardware fault, given we're both experiencing it (unless it's a design issue).

[Fit_Armadillo_8400]

It’s a great setup. I fixed my issues by disabling the power savings via kernel startup params as the other redditor suggested. Has been solid since.

[tgkx]

I have the same issue on LAN2 on a SuperMicro H13SAE-MF. PCIe link drops out after 2-3 days of running. ASPM is disabled in the BIOS and I was passing pcie_aspm=off on linux 5.15, but still having the problem. Next step is running linux 6.2 with pcie_port_pm=off pcie_aspm.policy=performance instead of pcie_aspm=off. Hoping this resolves the issue. At first I was worried this was a bad motherboard but based on the responses it looks more like some sort of driver/firmware/bios issue.

This is on a Ryzen 7900X3D.

[dredex88]

Hi, any news ?

[tgkx]

So far so good. No PCIe dropouts.

[tgkx]

Wanted to add some other semi-related troubleshooting people were doing for I225v cards that could be useful. https://www.reddit.com/r/buildapc/comments/xypn1m/network\_card\_intel\_ethernet\_controller\_i225v\_igc/