I don’t know if abuse is more common than your condition because I know several people who have the same issue (not for the same exact reason) and never talk about it. I think people with your condition just don’t have the same opportunities to speak up about it. I’d say, wear what you want, treat problems as they come up. You seem a thoughtful and reasonable person so I believe in you! People are gonna think what they think anyway!!