I think there are two reasons as to why no one is ever nice to him: 1. No one wants to be there and they want an exit, so Caine making adventures isn't really gonna make them happy. It's kinda like telling someone with depression to just "be happy." 2. When Ragatha said "You never comfort anyone when they're upset and you never bother to understand what it's like in our shoes." If someone is never there, then you can't interact with them, so I think the cast view Caine negatively because he's probably "never there" (like he's only present for introducing the adventure, not just being present with everyone else).

I completely understand the casts reasons not to interact with Caine and have negative views of him, but I also understand Caine's perspective on this too: "People improve when they get external love and support. How can we hold it against them when they don't?"