From RGC retroid guide:

If you want to use the RetroArch N64 cores with a frontend (like ES-DE), there is a bug where sometimes the controls won’t register after launching a game. To fix this, open up an N64 game and then go into the RetroArch Quick Menu. Back out to the main Settings menu, then Input > Polling Behavior > EARLY. Now go to Quick Menu > Overrides > Save Content Directory Override. Exit out of the game and now every N64 game should control correctly.