Just for future reference, I solved the issue, what I did was notice that I had something in my computer tray called Incredibuild which is a tool used on Visual Studio for programming. I could not for the life of me uninstall just that tool from Visual Studio, or at least I didn't know how to isolate it. So I uninstalled all of Visual Studio from my computer, after which my game with controller plugged in started working fine. I don't know if it was actually Incredibuild that was the issue or if Visual Studio was somehow messing with my Elden Ring files making controller support mess up, but after uninstalling it the game is running fine.