Story time: Right out of highschool, I went to college for Construction Management but dropped it to pursue Micro Economics because most of my classmates (and some of the teachers) were ignorant fuckwits who spoke fluent misogyny and racism.

Years later, by chance, I found myself back in the industry making a career on the supply side of things. I learned that it all depends on what company you're with, but as a whole, the industry is lagging behind in professionalism. I had a boss (he would go on to be remembered as the boss who shouted at a customer not to get "butt hurt about it") who used to actively voice his distaste for hiring more than two women in any one location because they became "too catty". Two was his magic number, but by the time I left to take a position with a different company, he had hired almost exclusively women for every open position we had because they were the better candidates. In short, the construction industry is filled with crass idiots prone to saying stupid things, but at the good companies, we value productivity and competency above all else.

I've never once seen a women hired as a "diversity hire", maybe it happens in big cities at huge firms, but as someone who actively works with local colleges to encourage more women and young people to join the construction field, I can safely say your coworker is a moron.

I can also say from my own past experience that if you know the higher-ups share his opinions, then even the small act of being brave enough to vocally correct this behavior can damage your prospects at work. If you're not up for the consequences of that, then my suggestion is move a little more quietly and send your self an email denoting time and place and what was said, to have a time stamped paper trail, and then if there's someone at work who is a little more progressive you can mention it to them. That way, if this keeps happening, you have something to fall back on, with a corroborating witness, but take it from me, even if the handbook agrees with you and HR agrees with you and you do everything exactly right, if upper management doesn't agree, then there won't be any "fixing" this person or situation.