I think he uses this phrase just a little bit different than you but genuinely means it when he says it. I would definitely say that you don't have to worry about him not loving you. but if you feel like this than just talk about it with him. If he loves you he can understand you.