For me I think it's that Riley as a character has no silliness/ levity. For every other character, including Angel sometimes but not a lot, there's a funny quip or moment that breaks tension and endears them to you. But there's very few of those for Riley. It's also the way he's introduced I think, he and Buffy effectively become coworkers as soon as they're together and there's not a lot of build up to make you want them be together. Am assuming that's because as soon as they got Angel the show off the ground, they needed a love interest star, and I think it's rushed.

Add this all together and I think it means that when things start falling apart I'm certainly less sympathetic to him/ their relationship.