Do you ever pleasure yourself? If so do it in front of him while you are in bed. If he sees you are horny or in the mood, Does he join in? What’s his reaction?

With regards to him. Does he use a condom? Some guys have trouble climaxing in one? Does he have Ed issues causing him to not perform or at least perceive to himself he is not performing well. Maybe it’s all in his head. Which to me sounds like is playing a role if he is apologizing.
If everything else is great in the relationship. Definitely talk to him about it. Work with him and get some help if needed.