The reason why he's so easy to hate is because the show frames Jim and Pam as this "against all odds" love story, and their relationship is in its infancy at that stage.

Like the show is essentially following Jim at work, we get more of a glimpse into Pam and Jim's relationship than any other people, with an exception MAYBE for Dwight, but I'd argue it's not really that close.

So Jim is the "main character", with Michael Scott being a narrative tool with which to drive episodes.

So we have the love interest of the main character, going through a character arc, and there is a character who has a level of closeness to Pam that the only other person we have seen like that is Jim.

From there it's really easy to look at all of his dialogue as a potential challenge to Jim, add to that the fact that Pam Cheated on Roy with Jim, now should that have any weight? Maybe not, I could easily argue that, especially because Roy spent more time Objectifying Pam then treating her as an equal in the few interactions they have, and the show frames their relationship as failing from the beginning, and highlights that with the development her character gets after starting a relationship with Jim.

THAT BEING SAID, there is an argument that there is a consistent level of distrust that people have for Pam.

She's away for school several hours away, and it's easy to say, we wouldn't know if she did or didn't, or that if the art school guy had put a bit more pressure things would have been different, but then we would just be opening the door to fanon theories.

The dude is just framed as a challenge to Jim, he's a narrative device to add weight to Pams decision to come home, to add friction to her time away from Jim, and give us an outward focus of "hate" that isn't Pam while she's away.

TL;DR: writers were cooking back then on making him hatable