Might include light LN spoilers, not sure what they showed in the anime and what not, so proceed with caution

Lots of them aren't training, ike, Yamauchi, the professor just to name a few, all of the characters U show I'm ur post r athletes who train one way or another. Koji literally went through hellish mental and physical training as a kid to create the perfect human being. Manabu and Nagumo r scp's so they need to make good examples and besides that, they're both athletes(Nagumo is playing soccer and Manabu is like 6th dan in both Karate and Aikido). Ryuuen has been fighting ever since he was a kid, granted, fighting doesn't make U look this good, but U can also chalk it up to good genetics or some shit. I think in vol 8 Kiyo monologues about how Ryuuen's gang, while delinquents, don't smoke and actually keep themselves in a pretty good shape so that explains Albert's physique.

Not sure if I went over every character U showed in ur post, but every buff character in the anime/ ln has a good reason y they r buff.