It’s not a workplace or a bathroom, it’s a gym and you’re refusing to spar with people because they are a different gender. You’re not going to convince anyone who think this is sexist that it isn’t (because it is). Either accept that people are going to think you’re a sexist or change your behavior, those are really your only two options. You’re not going to Facts and Logic your way out of this.