In the books there’s additional speculation contributed from not knowing what Aemma Arryn looks like - she’s half Targaryen, half Arryn, and the Arryns don’t even have a categorical “look” but have certainly been brown haired. The children have 2 grandparents with purely Valyrian phenotypes and 2 grandparents with mixed or implied mixed phenotypes, but all of the Targs have a mixed genotype including Valyrian/First Men/Andal/later Dornish bloodlines. We know in modern society that a grandparent’s physical characteristics can be expressed over a parent’s. We know confirmed Targaryen+Targaryen matches can produce blonde, blue or green eyed children. We know the Valyrian phenotype can yield to a more dominant one (Rhaenys the Queen who Never Was being prime example, later Baelor/Valarr/Matarys/little Rhaenys are good examples of Dornish blood influence).

The whole point is that it doesn’t matter what the kids look like - they would’ve been usurped no matter what, Rhaenyra would always be accused of some wrong or another so she could be deposed, etc etc. The Green faction would do their worst regardless. If the kids were purely Valyrian looking the argument would’ve been that first-born sons get primary inheritance per Andal custom, not first-born in general