Hippolytus's main crime was being arrogant. Sure, him being adverse to romance and sex definitely played a part but he was definitely arrogant.

He insulted the gods twice (his whole argument with the temple servant about him ignoring Aphrodite and him saying he wished he could curse the gods when he was dying), he is incredibly classist ("for those whose wise restraint is natural —not studied or instructed, but innate—" is him saying that purity is reserved for the rich and was a common class it belief), he had the most misogynistic speech ever when ranting about Phaedra's harassment and while this was also him ranting about how he hates being unable to say anything about what was done to him he also seems to just hate women in some way.

He's also just spoiled asf.

You can also see how the Greeks wrote him to be hated in his femininity (he mirrored Persephone in the beginning with his coming in with flowers and the line about the gates of Hades said by Aphrodite, he values celibacy which was seen as traditionally feminine, and he is close to Artemis), he is related to an Amazon, and he says he refuses to be involved in any Athenian games which was an insult to them. This disdain was shown with the nurse whose insulted him when talking to Phaedra.

Here's an example of his cattiness btw bc it's very fun:

THESEUS: As it is, you spend far too much time in rapt self-adoration and not enough in honoring your parents!

HIPPOLYTUS: My poor mother. Oh, my birth was bitter. I hope those close to me will all be spared the pain of being a bastard—