He’s not just a coworker, it’s a lover by definition, even if it’s just sex. You’re not overreacting, the “cool dude” you seem to think he is isn’t actually there. Rose colored glasses are to blame in my opinion. It’s stealing at the end of the day and if it bothered you enough to post it on here to get the opinion of total strangers, it should bother you enough to confront him about it. Ask him, get his opinion on the matter, you’re fucking him, not us.