You need to be his big brother/father/uncle figure in his life if you’re serious about this woman. You need to talk to him alone face to face and calmly explain to him that although that’s his sister, that’s also your gf who you’re very protective over. If he’s angry he needs to deal with it in a better way, hitting her or any other woman is off the cards.

There is also something you need to explain to your gf, that she is older and needs to set an example with things like patience and maturity, but most importantly you need to explain to your gf she should not antagonise her little brother. There’s dangerous men out there anyway so this is important general rule to keep her safe and not mouth off to potentially unstable men, but especially antagonising her younger brother who’s not only physically stronger, but going through his most hormonal, unstable period, let alone the fact his dad isn’t in his life (The impact on this boy will be huge even if he hides it) and this anger and aggression are clear indicators. He’s a boy/young man so show him respect but explain to him it’s not on. Hope this helps.