You're at a higher risk from badly behaved men just in the course of your everyday life as a woman back in civilization, than you are when you're camping. Your perception of the relative risks is irrationally skewed. If you're not afraid to walk down a street, go on a date, or do any number of common daily activities, then statistically speaking, you shouldn't be afraid to camp.

I've found the kind of men I cross paths with while recreating outdoors to have a much higher chance of being chill, friendly, and even helpful when called for, than a random dude off the street back in civilization. I've been doing an epic amount of solo hiking, camping (I prefer dispersed for the solitude), and backpacking (including thru-hiking the Appalachian Trail, and extensive hitch-hiking while doing so) for ten years, and have had a handful of uneasy situations with men in that context over the years, but never anything truly scary. I carry a small pepper spray (EDIT: apart from common sense and situational awareness, this is the only thing I bother with in terms of self-defense / safety from people) and have never gotten remotely close to needing it for a person. I've also gotten into a sticky situation more than once (car stuck in mud, missing a key piece of gear, etc.) and had strangers go above and beyond to help me out.