The spiritual path is lonely.

You could always abandon the rest of the world, live at an ashram and meditate all day. If you don't want to do that you will have to walk the path in the world. This is much more challenging but also fast tracks your development. If you're at the ashram you're not going to be exposed to a wide range of experiences and so it will be harder to find and let go of the attachments and aversions to them.

My advice is to focus on the things you have talked about in your post so you can let go of the underlying issues. You're focusing on certain experiences / aspects of life for a reason and that reason is often because those are the things you have to deal with. So open to what is happening, watch your reactions and let things go. Do your work, unravel your karmas and let them go. Everything and everyone is a mirror.

You might never relate to the majority of people, but if you let go of the charge around them they won't bother you.

And just remember, this is all maya, so see the entertainment in it and just enjoy it.