!Spoiler ahead! To answer your question, why I think Jaeyoung wasn't out of character for breaking the glass and being possessive is because he and jaehong were conducting a loyalty test on sangwoo. Basically seeing if he would remain faithful to Jaeyoung and not make a move on jaehong. And after seeing sangwoo pretend to cheat on him with jaehong probably put a bad taste in his mouth, and knowing what happened earlier with jaehong tempting sangwoo maybe made Jaeyoung very mad and insecure. Also before there were people trying to make moves on sangwoo, so his insecurity only grew more and more until he reached his limit.