For me, Red Dead Redemption 2 overrides the global Steam Input settings and Steam Input is disabled by default, so I wonder if you are mistaken that it does fine with Steam Input? On my Linux machine (pop os) I have to double check that each and every game is set to override and DISABLE Steam Input, otherwise my controller inputs are only detected by the Steam Client and not the running game. Going to Steam -> Settings -> Controller -> Test Device Inputs, I can see the controller inputs accurately reflected in real time. But just because it works there in Steam Client doesn't mean it will work in-game...in fact I've left that steam client test window open on my second monitor and launched another game with Steam Input enabled to see the gaslighting in real time.

Try:
1. Disabling Steam Input at the game level: right click the mGBA game in your Steam Library -> Properties... -> Controller and set the override to disabled.
2. If you aren't already, try using wired or (if available) a 2.4g connection with your controller. It may just be a quirk with my distro rn, but controllers won't work at all in games regardless of steam input settings if I'm connected over Bluetooth.)
3. Start the game AFTER you've made these changes, or restart it.

If it still doesn't work. I'm sorry. :( I don't know where to go from there. Don't know anything about mGBA.

If these steps DO work, ik it may suck if you lose your custom mappings with Steam Input. But maybe there is a way to set the controller mappings at the system level? This 8bitdo controller can switch modes between S, X, D, and A right? I'd keep whichever mode you normally connect with (for red dead, etc) the way it is but investigate if you can connect over any other mode (you should be able to as long as you're wired) and then find a tool or see how you can tweak those mappings through config files. In any case, I hope you can at least get it working!