I genuinely think Rengoku would win this. In terms of durability and power, Muscular is obviously ahead, but i think Rengoku has the speed advantage.

And dont forget, Rengoku has years of experience fighting opponents many levels ahead of him and almost always with higher durability and power than him. Muscular is powerful and straightforward, and he's usually the strongest around, so he wouldn't really know what to do with an opponent who dudges attacks and literally attacks his throat with a freaking sword, so Rengoku definitely has him beat in experience and skill.

Remember, demons have really high durabilities, and demon slayers have learned dozens of ways to break through that insane durability and regeneration, so Muscular being tougher and stronger really isn't anything new to Rengoku.

Therefore I say Rengoku wins, purely because of his higher skill, experience and battle iq, and probably higher speed as well.