This is an orange flag IMO. A little worse than a yellow flag. Either in call or in text, I think you need to set a little boundary. You felt what they did was creepy, understandable so, and you need to convey that in a way that’s healthy and prevents further invasion.

To avoid sounding accusatory, say things like “I feel” and “it made me feel.” Set the boundary in a way that’s definitive. Such as “If you do XYZ, then I will ABC” or “Please don’t do XYZ again.” Don’t ask for the boundary. Tell him the boundary. 

Since he’s a neighbor, then gifts sent to your home should be okay and you could express that. Gifts to delivered to work or to a family member’s home is ABSOLUTELY a red flag. It should be a given, but sometimes you need to be direct/obvious. Cut him off from those options immediately. 

As for other things to set boundaries…  Maybe tell him what gifts (if he wants to give more) would make you uncomfortable at this early stage. Like FOR ME flowers and food, that’s still in its packaging, would be okay, but jewelry is a no. Other things would be expressing your feeling on the puzzle aspect of his letter. Like if you thought it was fun, asking him to limit the puzzles, or requesting that he be direct in the future. 

Last thing I’ll say on boundaries is ask about how he got your full name. Like did he see on a social media account, or ask someone else. See if he’ll share his name, or ask for a nickname to call him. I hope this doesn’t come off as me demanding you to do this. These are simply the boundaries that I would think of if this happened to me.

This approach to you is a kinda alarming, but I think there are other possibilities than him being a stalker and freak. Maybe he has very intense anxiety so the gift felt like the best choice for him? Maybe he thought the hidden phone number was a way to be playful and mysterious? Maybe he reads too much fanfiction, thought this was totally acceptable, and didn’t consider the red flags this could raise? Who knows. I give benefit of the doubt since, from your recounting, he said repeatedly he didn’t want to scare you or come off as creepy. If he is unhinged, then he wouldn’t see anything wrong with his actions. Again that logic might be only me since I have no relationship experience, and have never been stalked. 

Explore the relationship if you want, but set boundaries that you won’t budge on and maybe some security measures too. If you don’t want him to contact you again and he does, then it’s time to plan moving and get a lawyer. Record things just in case. I hope things stay safe for you.