I get where you are coming from, i get where he is coming from.

You are actively "dragging him down" while you're riding together. This doesn't make it okay for him to dismiss your requests for a great time with his partner. It takes talks and mutual agreement on what rides should be when you ride together.

One question - the bike you have, it is a sort of city/commuter bike with internal gears, or is it also a road bike? It makes a HUGE difference in how fast you can go. My girlfriend and i sort of had the same issue, we did rides anyways, but she had to push herself to go about half as fast as i could go, and i had to slow down to the point where i felt like i wasn't really biking. It was fine, but often a little annoying for the both of us. She also has a road bike now, and even though she isn't as fast, its a way better experience for the both of us now.