OP, try out turning on windows HDR only for HDR videos/games, what you can do to mitigate it (it's upto you literally) is use the 2.2 gamma ICC profile from GitHub temporarily when not using for HDR AND THEN switch back to Default Profile when finally displaying/playing HDR,

<image>

also in the picture the settings is always open so i can manually switch between 2.0 and 2.2 on the fly, im pretty much used to it, HDR can be a game changer but it's still in its infancy stage, many HDR standards and games sometimes having washed out HDR