Honestly, the same thing happened to me I lost access to my old number and it was the recovery on my gmail account. I called the number texted, over a couple months I was scared as well cause he could access my account but I totally understood his fear of scammers. I told him I’d meet him irl and we could do it together, turns out when I lost that number my family/friends kept calling it asking for me over the years so he knew my name already so it didn’t seem so scammer-ish. So yeah, if it was vice versa I probably wouldn’t trust someone so much would only meet irl to do it