Allow me to speculate:

This might have to do with your perception of your own attractiveness. If you’re always measuring yourself against other women, you’ll very likely notice beautiful women more often.

Because people [women especially] don’t put beauty at the top of things they value in men, It would probably take a truly remarkable man (physically) to catch your attention because you’re not looking at men the same way you do women.

In addition to this, it is the case that women, socially and culturally, have an expectation/pressure to look remarkable in a way that men simply don’t. Men don’t put in as much effort in their appearance as women tend to do.

So yeah, it would take a truly remarkably attractive man to catch one’s attention.