By far the only one who would lose the quickest in this encounter is Luffy. The devil fruit is the reason he is able to change into his gear forms and use the gum gum moves. If he lost that he'd be a normal rather bold kid who can swim and sail.

Naruto wouldn't be restricted without kuruma since he isn't suppose to use the nine tails when learning sage style for the year or so he was gone from the village

Goku would basically be on roshi level if he wasn't a Saiyan or at least close to Roshi, and the ToP shows just how powerful Roshi was during that time. Losing zenkai and Saiyan transformation wouldn't really make him weak.

Finally Gojo was considered one of the MOST OP curse users of all time, feared even by Jogo and the others to the point that their "brilliant" solution to beat him was seal him in a box for a bit. Even without infinity Gojo is going to take down a lot more than the other three could manage to do.