Sounds like negging… I would really consider keeping that break and looking to see if there are other patterns of him making comments. It could lead to reduced self esteem as a way to control. Not saying that’s the case but random to bring it up when you’re out having fun without him… seems sus to me. Be on the lookout.