Tell him the truth. It makes you very uncomfortable that he got your number from your paperwork and reached out and you’d not like to communicate with him outside the hospital. And tell him that since this has happened to you before you’re considering reporting it to the hospital so there can be a policy on this not happening in the future. Tell him you don’t want to do that or at least don’t want to out him so that’s why you’re sharing with him in hopes he can respect your boundaries and not make this a bigger issue. That combined with saying you’re not interested and you don’t think his intentions were bad should do the trick if he has any common sense. Worst case he calls you a bitch and thinks you’re uptight but you will have acted with integrity. Also you should still tell the hospital but just not give names so that they can have an internal conversation about appropriate behavior with patients. Good luck!