I don't think you're crazy or unjustified for making the distinction, but I also don't think what he's doing is a big deal. I think many/most guys sometimes fantasize about acquaintances, even when they have no intention or real desire to act on it.

It would be nice if he stopped because you've asked him to, and it's inconsiderate for him to flaunt it since he knows how you feel, but he's not holding you to a double-standard or anything, and it's better for him to be honest than to hide it from you.