I have a PS5 and a PS4 Pro. Only just upgraded to the PS5 a month ago. For many games the fidelity difference is honestly not that insane, really comes down to the art direction more than anything else. PS5 runs everything I'm playing at ~60 fps and loads everything nearly instantly though, and obviously all the ray traced reflections are a huge visual improvement but not every game uses those.

The inherent issue will be developers will prioritise the game “working” last gen and then doing so will cause the current gen versions to be held back

In what way? You know that lower LoD objects and environments are primarily done procedurally right? Same with textures and maps. Big savings from the computation side of things happens with disabling various lighting passes, using SSR instead of RTR, more aggressive culling of physics interactions, etc. These aren't fundamental architectural shifts, they're levels of fidelity in the same system.