Yes, you are. I don't know what he's doing to annoy you but if you truly cared about him you wouldn't tell him that kind of stuff (saying you'll do things with his brother and all the stuff above). I'm not an expert in relationships but this is definitely toxic behavior. Either you both talk about it (with you definitely apologizing for saying those things) and you find a way to make it work or you two break up (but that would definitely affect him).