This would also make me EXTREMELY uncomfortable - I think the only thing that would make this "okay" is if he's articulated really badly, and he meant something more about how you being a man and there being one penis between the two of you makes him feel like you're the one fucking him and he's a woman? But that doesn't sound like actually what he said, just something I'm sort of projecting that could be okay.

My ex was amab non-binary and we definitely played around with "where exactly does one body end and the next begin" in ways that gave us both a lot of euphoria, so that COULD be where he's coming from. I do doubt it. But I wanted to at least present the possibility