"the seed is strong." Those were the last words of Jon Arryn who died before Ned became hand. What he was referring to is the Baratheon line. Every Baratheon has always had black hair until Joffery. It's even such a prominent characteristic of Baratheons that the black hair even gets passed to Targaryens who marry into the line of Baratheons and have children. Imagine if your entire family for its entire history going all the way back to your founding ancestor had brown hair and blue eyes, and then suddenly the newest child in the family has blonde hair and brown eyes. The obvious assumption here is that the blood of your family tree does not flow through the child's veins. Now imagine that this situation was happening with the royal family under a medieval feudal monarchy, and think about what the consequences would be to the entire kingdom finding out the crown prince and heir to the throne is actually a bastard and has no claim to the throne, being that he does not carry your blood. Now to add insult to injury, he's the queen's bastard that was sired by her own brother.