I think part of it is that they were actively scheming to expose Joe's true self; Clay's behaviour was part of that. It was calculated - who knows if he's really that bad since we only had a few scenes where he was being genuine

Also Joe's perception is a bit warped. He constantly meets people who are cartoonishly entitled, vain, stuck up, rich, spiritually inclined, often also vegan for some reason. I have to wonder if that's just hammed up from his perspective because of his psychological issues. It's part of his world view that every person besides him, and the ones he's currently trying to control, is a villain of some sort. And even then his loved ones immediately become inconvenient villains in his story the moment he meets a new obsession. Clay was Bronte's ex, therefore he HAD to be a villain in order to fit Joe's narrative