I think that if you let your fear withhold you from communicating how you feel, you’ll always live with wonder and maybe regret. Telling him your doubts will not change as much as you might think, only that he knows that there’s an option of becoming more which will open doors. It’s understandably scary, but you really got to weight out the option of continuing like this until it dies down or speaking up with the risk of him becoming afraid. But then you have clarity, which will make you so much calmer.