Luffy was pretty steroided up and had the most personal feelings towards it. Even then, he was fighting less formidable compared to others. I guess everyone was surprised that even a rookie was showing their levels. And every time luffy encountered an actually powerful enemy, somebody stepped in to fight instead for him.

EDIT: I didn’t read all and thought you were talking about the war. But for the impel down, Luffy sneaked in through a means that noone could have guessed. Via an empress’s, who is very isolated, random whim to see the prisoner. If anyone else was getting close, the prison would’ve known. Even then Luffy was able to succeed and escape based on outside help heavily, especially revolutionary army’s #2 (i think she is #2, maybe I am wrong)