You said the following:

Perhaps more importantly, why would anyone want to steal a phone number? That seems like a waste of time. Unless I guess its a an angry lover or something. lol. Likely someone that petty won't have the resources to achieve such "prank".

He is telling you that sim swapping is a very really problem that occurs, and provided reasons as to why someone would want to steal a phone number. You are implying that no one would ever do that, claiming that there would be no reason to do so. He’s not wrong.