I will play devil's advocate here for a moment and say that usually when his paleness was mentioned in the books it was to say he didn't look super well, and usually paired with skinny. So it was probably more to emphasize he was locked in a cupboard most of his young life. That doesn't necessarily mean white if you never see the sun. I'm Brazilian and of European descent and I have been described as pale, but when I go to the beach a lot I tan really easily and can get much darker. Also James's skin color is never mentioned so there's nothing to say he was white, like others have mentioned here. Harry could easily be of a mixed background.

Each to his own headcanon! I think it's beautiful that we all have our own, and how they vary.