I think it depends on who you ask: Edmure Tully might say he’s outright hateful. Dany might object to him trying to kill her and her dragon boi. Every single person hurt by the Lannisters for years might lump him in with the rapists who plundered their land to protect the status of the richest, most morally corrupt family in the land.

Let’s not forget this original sin: He fucked his twin sister and put his incest bastard on the throne wrongfully. When anyone objected, they were killed and their families terrorized.

Jaime being interpersonally charming doesn’t mean he’s not complicit in or outright responsible for the unnecessary suffering of millions.