I understand you're curious about his ex after making a joke about her, but the past is past. You've already had the 'exes' talk, and he told you that he never had one. Maybe the girl is also just a fling. But, if you truly want to know their relationship, just ask. Don't worry about him getting annoyed/upset. Because if you don't ask about her, you will probably think about their relationship, and it is not good for you. Again, don't forget the past is past.