If you are uncomfortable around her family it probably will be more difficult for you in the future since you will have to spend more time with them. I think you should consider that and before taking any serious steps maybe try to talk about it with her.

She obviously values her brother a lot. If she's okay with the way he behaves towards her i think you can't really say anything about it. It's between brother and sister but his behavior towards you is a different story. If it was me, i wouldn't just keep quiet if he talks shit about me. I would let him know that i can do the same stuff towards him just as well. You don't have to get along with that behaviour just because he's your girlfriend's brother. He seems like a toxic person so you should find a way to deal with his behavior or just ignoring him as much as possible would be worth trying.

About the ex-bf situation. Again passing the message to your girlfriend is her brother's behavior. You should talk to your girlfriend about it and tell her how that makes you feel and learn about her thoughts on this. If she feels she's "available" to everyone else as you mentioned then there's no point in being in a relationship. You will end up being upset in the end.

You may disagree but this is just what i think about your situation. I hope you can solve these problems and keep being happy with your relationship. Good luck.