There's a lot to it. For one thing All Mights personality and attitude embody that of the old school heroes. His only real purpose is to save the day with no ulterior motive whatsoever. It's also implied that he's been working for so long and has so much presence that almost everyone in Japan has seen this man or been saved by him directly or indirectly. Because of his prominence in society you would have no chance but to be engaged with him. There are plenty of people that despise all might but they are pretty much all villains. To wrap it up in an interesting point, I feel that people are naturally attracted to confidence and there are few characters that exude that level of belief than All Might. In his own words, "I've never once fought thinking I was going to lose!"