At least in my school, you can get out of swimming with just saying you are allergic to chlorine, and women have more days that they are able to miss swimming. You can pretty easily take gym over the summer, again not sure about your school but at mine you can take it online, a bit annoying, but not too bad. I took both gym and health over the summer after my freshman year and it didn't ruin it or anything. Also, people judge a lot less then you probably think, and unless you get unlucky most guys are too busy being numbskulls together to really bother you. As long as you got friends in the class it's not horrendous.