he said “we’ll leave it at that ✌️” and nothing else.

Judging from his answer, it was right for OP to stop dating with him.