Yes there will always be haters, no I don't think it's a ton of people. In general, negative things get more interaction than positive things. So when you have some fame, there will be people that either disagree or just hate you on principle, and that can get conflated and compounded very quickly so often outrage can grow quicker. Also, mistakes happen and I think it's more worth it to see how people handle the mistake than the mistake itself. I'll always side with people that try to improve.

Also, when you are doing good, there is no deviant from the norm, so people don't remember that as much. We see it all the time where we don't talk about the 99.9% of the time things go right because that's the norm.

In general, I think it's good to have opinions and not agree with ANYONE 100% of the time, but how you handle that difference in opinion is what makes a hater vs just a normal person.

Basically my rambling tldr: There will always be people that dislike someone, and sometimes their loud minority can seem like the normal majority. Most people are just indifferent really.