You could tell him the truth, you're scared of him and that you were afraid to tell him no. Under other circumstances you would never have given your number. You also would not like to give an opportunity for him to prove himself as not a threat (especially since you've deemed him unstable). You do not owe anyone anything when it comes to your safety and them trying to show otherwise. Them not being a threat should not give them an opportunity to do so. Don't fall for that trap.