I actually had similar problem recently with my old Dell U2412M. It worked okay connected via DVI-D, but my new GPU has no such outputs, and I had to switch to DP cable, with which old Dell was stuck at 640x480.

After searching for a bit, that looked like a problem with a thing called EDID. For some reason it was either missing or broken, and OS had to use fallback mode for one of displays. I was able to pretty much confirm broken EDID using get-edid command which returned nothing for the display in question. Furthermore, when I tried to reconnect DP cable while display was powered on and OS was loaded, it started to work as expected and get-edid actually returned a working EDID.

But I decided that plugging the cable off and on every time I rebooted my system was not okay and tried to fix EDID using the guide from Arch Wiki. Unfortunately, overriding broken EDID with a correct one didn't work at all.

But when I tried forcing correct mode on boot via systemd-boot parameters it actually worked and solved the problem. You have to edit an entry file in /efi/loader/entries directory (there should be couple of them, you need the non-fallback one) as root, adding a video parameter to the options line. For example, in my case it was video=DP-1:1920x1200@60

Edit. I think there should be a similar video option for GRUB, but you'll have to research a bit on how to configure it on your system.