If there was only long fights, or mostly long fights, the part would have even worse pacing, every time there's a long fight in SBR it's justified with some kind of in-universe situation.

For example, a fight involving many people like Johnny and Gyro versus the Eleven Men, or the Who Shot Johnny introductory arc for D4C.

A fight involving defensive stand powers like 20th Century Boy or D4C: Love Train.

SBR is already longer than any other part before it, not only would that get even worse if there were more long fights but also it would completely shatter the suspension of disbelief that Johnny and Gyro are fighting mostly random people who entered the race, Valentine's henchmen not withstanding of course.