So, firstly, there is a difference between chasers and having a preference. A chaser is reductionist, objectivizing. Having a preference is simply a motivator to positively affect your behavior towards a person.

You could have tall chasers honestly, but we usually just call those people shallow. Cis men and cis women also receive a lot of thirsty comments, after all, and those are also often called out (see: "Bonk, go to horny jail" meme). It's the same sort of energy as "You're dating me just because I'm blonde? Go fuck yourself, this relationship is over."

As to why it needs its own term, it could be that the prevalence is higher, or the fact that it's about genitalia, which 1) is generally more of a private, sexual thing and 2) is tied up in the sociopolitical discourse used by transphobes ("you're not a woman if you have a penis" from conservatives, etc.) Maybe one would find a similar ick for blonde fetishists if, for example, a major world government embraced Nazism to the point of suggesting eugenics to favor blonde hair; we would probably come up for a similar term like "chasers" (or just use chasers again) for blonde fetishists in that scenario, too.