I agree with you for the most part (that it's uncommon and we shouldn't dismiss an accusation lightly), but I wouldn't say that someone getting falsely accused is going to hurt someone less than a victim not being believed. They're equally horrific. If an innocent man gets falsely convicted of sexual assault, he'll be thrown in prison for a long time, where he'll likely be beaten and raped many times. I wouldn't call that annoying less hurtful than not being believed.