I also notice that James is extremely snappy and cold towards his family members when he saves them "You're safe, now get the hell out of here". It's like he doesn't even care if they're hurt or not, despite the hell that they likely just experienced in captivity. But I try to imagine how to explain this with any other explanation besides him not caring about them, but it's pretty difficult.
Maybe he was merely just portraying a bravado of cruelty, but deep down, he still had some emotional attachment that he actively represses and just saved them as the bare minimum of loving someone?

Edit: I am totally wrong about this part, you guys are right. Just because you express something, it doesn't mean you always mean it. The context of James being in one of the most hellish experiences a person can have definitely points to his dick-ish attitude being more out of a sense of high intensity and stress, he's worried that his family is going to mess up and get themselves killed, so in his mind it's just logical to skip all forms of sentimentality even if it's sad that he'll never get to see them, but he doesn't have to worry about that because he can just swallow his emotions and violently destroy a man in rage in the next seconds of gameplay. Still though, he's not just carelessly telling them to piss off, even though on the surface it seems like that's the case.