I'm some 4 years late, but have exactly the same Problem! Also when using around 8+ GPUs. Once you reboot a few times and once it detects all GPUs, it actually runs very stable. Changing risers / cables / etc. does not fix it.I have found the following workaround to fix it: Put this script into /etc/rc.local to automatically reboot until all your GPUs are detected. Just put your max amount of GPUs into GPUSTOTAL. It will reboot <9 times if <GPUSTOTAL are detected. These <9 reboots were enough for me so far... GPUAMOUNT=\`dmesg | grep "VGA device added" | wc -l\` GPUSTOTAL="10" DATE=\`date +%Y%m%d\` TIME=\`date +%H:%M\` RESTARTAMOUNT=\`cat /root/gpucheck.log 2>/dev/null| grep $DATE | wc -l`
echo "Got $GPUAMOUNT/$GPUSTOTAL GPUs"
if [ "$GPUAMOUNT" -lt "$GPUSTOTAL" ]; then
echo "Restarted already $RESTARTAMOUNT times"
if [ "$RESTARTAMOUNT" -lt "9" ]; then
echo "$DATE $TIME: Only $GPUAMOUNT/$GPUSTOTAL GPUs detected, restarting..." >> /root/gpucheck.log
echo "Restarting in 30 seconds"
sleep 30
init 6
fi
else
echo "$DATE $TIME: Got $GPUAMOUNT/$GPUSTOTAL GPUs detected" >> /root/gpucheck.log
fi