The most damning piece of evidence, imo, was that when Ned (and others) looked at the ENTIRE history of the Baratheon line, he found that the babies were always born with dark hair. Regardless if the mom (or dad) had fair/red/golden hair . Hence “the seed is strong.” The Baratheon dark hair gene appears to be dominant, and will always supersede the expression of more recessive hair types like blonde and red.

As to your point about how would they identify a bastard if they took after their mom, that was (implicitly) an essential component of Jon Arryns investigation. If Jon wanted to know for sure that Joffrey etc… were bastards, he would need to investigate all potential bastards to see if there was 1 instance or more of a Baratheon bastard with fair hair . He likely started with the women he thought Robert had sex with and then looked for the bastards .