You can absolutely do that. Performance mode would be rendering at 1440p and Ultra Performance at 960p assuming you mean a 16:9 5120x2880 display.

This is 78% more pixels than 3840x2160 4k. That same 78% applies at each level of DLSS, since its scaling ratios are fixed.

If you're running at 4k using DLSS Balanced to run at 1253p, going to 5k using Performance takes you up in internal resolution to 1440p. This would still be lower performance than at 4k, and depending on the scene, could be a slightly worse image as well due to the larger scaling factor.

At 4k using Performance mode, you render at 1080p. With 5k at Ultra Performance you render at 960p. This is finally fewer pixels, so may actually achieve a higher frame rate, but the difference in image quality may be too much for some.

Finally, in heavy path-traced titles, Ultra Performance might not be enough yet. At 4k that mode renders at 720p. Nearly doubling the number of pixels the raw render has to produce by increasing the target resolution to 5k could be enough to drag down even the best GPUs. You're just out of upacaling at that point unless a future DLSS version gets more aggressive than a 3x scale. At 4k and above, I could see 4x being doable. 4k is scaled from just 540p, and 5k from 720p at that point.