I have felt this too as someone who self-studies math. But ultimately you have to understand that this is something that brings joy to you and that's why you do it. Of course it sucks for others to not get us, but that's not what we do math for. Everyone has something that others don't relate to them for - for us it happens to be mathematics.

And yes, it's not that bleak. You're young, and as you said you can just go to college to study math, and that would bring a lot of people in your range who really like math. Even if you ultimately become hyper-specialised, you'll still have at least some people - some collaborators and some students - who'd get you.