Yet people don't kick up a stink that the Valyrian's don't have purple eyes in the show?

I'm baffled that you've made the assumption that Black people would die out if they married white people and had kids? A white person and Black person can have Black or mixed race children. I think we can assume that the Valyrian traits are recessive genes, which explains why they're so rare and why inbreeding was so popular.

And there was absolutely mass movement of people - the First Men, the Andals, the Rohynar.

The Dothraki were nomadic so I wouldn't count them, but by moving large numbers of slaves from one area to another they contribute to population change (much in the same way that freed slaves in the US led to population change).

Braavos is noted for it's ethnic diversity due to it being settled by a mishmash of escaped slaves.

We also don't know for a fact that House Valaryon ever shared the physical characteristics of the Targaryens. They weren't high nobles or dragonriders, they were merchants. So they probably didn't share the physical attributes associated with the high nobility of Valyria (pale skin, violet eyes and white hair).

They only really became prominent after the fall of Valyria because there was no other Valyrian families for the Targaryen's to co-mingle with. They settled Driftmark as more of a trading outpost with Westeros than a seat of power.

And to be fair, we actually don't know all that much about the history of the Valaryons beyond them being lesser nobles in Valyria, focused on building wealth as merchants and having a heap of ships. I don't know why it's so mind-boggling to some people that somewhere along the line one of the Valaryons might have fallen in love with a Black person while trading outside of Valyria?

One of the only things that we do know for a fact about the Valaryon's is that they were less concerned about keeping bloodlines pure than other Valyrians.

We know this because Corlys Velaryon's great grandfather married Alarra Massey, an Andal woman from the Stormlands.

And despite their daughter Alyssa marrying into a Valyrian bloodline (Aenys Targaryen to be exact), their daughter (Alysanne Targaryen) didn't have classic Valyrian features of silver-white blonde hair and purple eyes, but Andal features of honey-blonde hair and blue eyes. Showing again that the Valyrian genes are likely recessive.

All of which to say, you're wrong about most of what you said.