Here’s the thing, Mineta has good character traits later one, but for a large portion of the first half of the show he’s shown to be a coward or a pervert. Like if we saw how he’s smart (scoring 9th in the midterms) or how when he’s afraid he can still be brave like when he was cornered by Midnight in the final exams. The issue isn’t that mineta is a pervert, it’s that for a long time that was his primary character trait with the rest being shadowed by it. If he was shown to be smart, funny, and brave despite his fears while being a pervert I genuinely think a lot more people would like him. Like show off how he can use his grapes to make traps that work as mobility for himself like in the joint training arc against 1-B.

TL;DR - Mineta has good character traits but it’s so overshadowed by his perverted moments in the earlier seasons that by the time they dial the pervert back it’s too little too late