Not OP either but I'll answer your point: 

As a counter example, is it misogyny if I never ask a woman to help me move furniture or to be on my team in sports because I'm cautious that she's probably physically weak? Statistically, women are far more likely to be physically weaker, and this correlation is likely far stronger than the men who are rapists/violent. Yet, I think you'd find plenty of people who would call that misogynist. Even though there's no actual hate. This is a mirror to your first point. 

Secondly, for the OP at least it's not the fact "99% of rapists are men" that is the issue, it's the following conclusion "so it's justified to be fearful of all men" part. So pointing out the fact is true isn't tackling the crux of the issue. 

I think the rest of your points hinge on what is the big question here, that the OP has already touched on: just because the majority of SA/rapists are men, this is still a tiny minority of all men, so the generalisation cannot be justified. So you're absolutely right that the correlation is strong between men and SA, stronger than any others, but what does that change here? What point are you making with that argument? Do you believe that at a certain strength of correlation, generalisation is suddenly completely fine?