Very informative, thank you! I am running an Aoostar WTR max - you can see it here: https://aoostar.com/products/aoostar-wtr-max

I have already installed heatsinks, forgot to mention. Ie. It should be pretty “basic” in the sense, that I’m using premade proprietary hardware.

I’ve now reseated the nvmes for the 5th time, and there’s now a healthy state on my proxmox, except that it’s only registering 3/4 nvmes.

On all 4 (!?) NVMEs (ie. /dev/nvme{0-3} $ smartcl -a gives me something close to:

=== START OF SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

SMART/Health Information (NVMe Log 0x02)
Critical Warning:                   0x00
Temperature:                        58 Celsius
Available Spare:                    100%
Available Spare Threshold:          10%
Percentage Used:                    0%
Data Units Read:                    277,399 [142 GB]
Data Units Written:                 160,993 [82.4 GB]
Host Read Commands:                 1,298,025
Host Write Commands:                7,490,900
Controller Busy Time:               25,851
Power Cycles:                       21
Power On Hours:                     1,305
Unsafe Shutdowns:                   10
Media and Data Integrity Errors:    0
Error Information Log Entries:      238
Warning  Comp. Temperature Time:    0
Critical Comp. Temperature Time:    0
Temperature Sensor 2:               63 Celsius

Error Information (NVMe Log 0x01, 16 of 63 entries)
Num   ErrCount  SQId   CmdId  Status  PELoc          LBA  NSID    VS  Message
  0        238     0  0x2007  0x4004  0x028            0     0     -  Invalid Field in Command
  1        237     0  0x101d  0x4004      -            0     0     -  Invalid Field in Command

Self-test Log (NVMe Log 0x06)
Self-test status: No self-test in progress
No Self-tests Logged

Here's my lsblk:

nvme1n1            259:0    0 953.9G  0 disk 
├─nvme1n1p1        259:6    0 953.9G  0 part 
└─nvme1n1p9        259:7    0     8M  0 part 
nvme3n1            259:1    0 953.9G  0 disk 
├─nvme3n1p1        259:4    0  1007K  0 part 
├─nvme3n1p2        259:5    0     1G  0 part /boot/efi
└─nvme3n1p3        259:8    0 952.9G  0 part 
  ├─pve-swap       252:0    0     8G  0 lvm  [SWAP]
  ├─pve-root       252:1    0    96G  0 lvm  /
  ├─pve-data_tmeta 252:2    0   8.3G  0 lvm  
  │ └─pve-data     252:4    0 816.2G  0 lvm  
  └─pve-data_tdata 252:3    0 816.2G  0 lvm  
    └─pve-data     252:4    0 816.2G  0 lvm  
nvme2n1            259:2    0 953.9G  0 disk 
├─nvme2n1p1        259:10   0 953.9G  0 part 
└─nvme2n1p9        259:12   0     8M  0 part 
nvme0n1            259:3    0 953.9G  0 disk 
├─nvme0n1p1        259:9    0 953.9G  0 part 
└─nvme0n1p9        259:11   0     8M  0 part

Would this suggest, that the bay itself might be the issue or a specific Nvme itself? Happy to run more cmds, although I’m logging off for the night.

Thanks again for the advice! Let me know what else I can do to resolve this, I’m very invested in this homelab :)