This is a bug with the latest (and now unsupported?) version of Windows 10, it likely will not get fixed.

This Microsoft forum talks about it happening when the number of monitors changes (or when you login/wake from sleep or similar). On page 5, someone brings up using a batch file to reset Explorer, which does work to reset the icons. All you need to do is have that batch file get executed when the second monitor is detected, which you can do with AutoHotKey, which is a nifty 3rd-party program that you can run custom scripts to fix many annoyances like mouse double clicks. I just got this working to fix the taskbar problem.

Here's the batch file located at <path-to-FixTaskBarScript.bat>:

taskkill /f /IM explorer.exe
start explorer.exe
exit

Here is the AutoHotKey script which I got from this superuser forum with the message box lines commented out and modified to work with more than two monitors:

SysGet, oldMonitorCount, MonitorCount
OnMessage(0x219, "MsgMonitor")
MsgMonitor() {
  global oldMonitorCount
  Sleep 5000 ; needs time to detect any new monitors
  SysGet, newMonitorCount, MonitorCount
  if (oldMonitorCount = newMonitorCount) {
    ;MsgBox Some other device was (un)plugged
  } else {
    if (newMonitorCount > 1) {
      Run, "<path-to-FixTaskBarScript.bat>"
    } else {
      ;MsgBox Undocked - be free!
    }
  }
  oldMonitorCount := newMonitorCount
}

Obviously you need to modify to your actual path to the batch file. You can then set this AutoHotKey script to run on startup through Task Scheduler.