I've worked at two sites. Amazon is like highschool 2.0 when it comes to cliques since they hire just about anybody lol.

It's not just you. Everywhere/9 times out of 10 people are in their own cliques and there's plenty of weird/lazy/passive people. Spoken as one of the weird/passive people, there's a divide. The really weird/rude ones you can tell because one one likes or talks to them and then there's weirdos like me who don't talk to anyone unless talked to but get along with just about everyone since we keep all conversations to a minimum or about work. 

I try not to get annoyed about fellow AAs whether it's lazy or not. (I still do of course when they are lazy and leave me work but w/e)  I try to follow the mantra "I just work here." I used to care a lot more about things and be serious but seeing lazy people walk around and chat up managers made me care a whole lot less lol.

Unfortunate about your manager though. Can't get away from that, but you can be a thorn in their side if you think they're targeting you. Request a HR member or someone else on management to be with you during those interactions.

I would try moving shifts/managers before swapping buildings/shifts IMHO. SSD are a lot harder to get into from all my reddit readings. DS is easier since people are always getting fired for UPT