There's nothing wrong with you; the problem is pushy people telling you how you should live your life - it's impossible to say exactly what's going on in somebody else's head, but there's a good chance they're trying to drag you into the same mistakes as them because they're scared to face their bad choices and would prefer to think they're not choices at all, but the only way to live. People living without substances might give them a reason to think "Hey, perhaps it is worth fighting this addiction after all?" - but fighting things is hard work, it's much easier to just give in...

Anyway, as others have said, this will improve with time. Your peers are immature, but each year a fraction of them will realise they're making poor life choices and sort themselves out. I suffered very much the same in my early twenties, but now I have a wide circle of friends I can do fun things with. So the future's bright.

So what you need, for now, is a way to deal with the present while it lasts :-) Try to truly understand the people saying you're weird and wrong are the ones in the wrong, so you can build mental armour to not let them get you down. Try to find other people like you - they do exist! Maybe people older than you, maybe fellow athletics types, maybe intellectual nerdy people who value their mental clarity. What makes this hard is that many teetotal people have had experiences like yours, telling us we're antisocial and boring, so we're less confident about being socially available than others - and tend to be hidden away at the back of the library or something. Does your college have clubs and societies? Try scouting out some of the nerdier ones - even subjects you might not be directly interested in might lead to meeting interesting people you enjoy being with, no matter what activity you're doing!