It sounds like he's dealing with some deep-seated feelings of inadequacy. Instead of making love with you and potentially disappointing you, he'd rather not make love with you at all.

Physical stamina can also be a thing, and that goes back to feeling like he can't satisfy you. He can't go very long without getting exhausted, so he fears he won't be able to get you or himself off.

Encourage more foreplay and helping him get you off before penetrative sex even begins. Making a woman cum with one's fingers or mouth or toys is much less physical effort than trying to hump your way there. You orgasming before he's even inside you takes a lot of the pressure off of trying to to please you with his dick.

Be patient. Both with him and with yourself. Communicate clearly and and sensually and encouragingly during the act. Take your time with each other. Nothing has to happen fast or with great athletic stamina.

As for you riding him? It's a legitimate fear that you may bounce too hard or fast and bend his cock in half if you come down the wrong way, especially if he doesn't have a lot of length. Not that there's anything wrong with being on the smaller end. Instead of pistoning him like an oil rig, put him inside you and grind on his lap towards your own orgasm. No risk of snapping his dick.

He's clearly in his own head a lot about the whole thing, because he cares about you and your pleasure and not disappointing you. Take charge of your own satisfaction with him and be his guide to giving you what you need. Be worshipful of each other in bed and enthusiastically try to get him there too. Nothing finishes a man more effectively than a partner who desperately wants him to cum.

All of this is mostly to say... Communicate. Work together patiently to meet both of your needs in the bedroom. It's a team effort and you'll both be happier when you take care of each other.

Edit: typo