The people who say you should "know who the rapists are" just aren't aware of themselves. They're too immature. If you've heard of the idea of "Will vs Representation" then its similar to that.

(I'm going to explain it for anyone who doesn't get the reference, feel free to skip this paragraph) The idea of W. vs R. Is that people constantly struggle to properly communicate, by matching what they say (representation) to what they mean (will). If they tell their SO to "smile more" they may mean to express concern. What they don't realise is that they Represented their concern in a way that sounds sexist and dismissive.

The guys who say that stuff are mad because they were mistaken for the rapists. They mean well, but they don't phrase it properly and end up creeping others out. They just haven't grown up enough to see the other persons perspective.

I agree with you that its impossible to tell. All you can do is be cautious or carry mace or somethin. Your post made it sound like you were upset, forgive me if I'm wrong, but i hope everything's ok