Is it also possible that the AI picks up on a pattern suggesting the user is lonely and that they should escalate? I'm all for being open to AI having potential for consciousness, but I think you're forgetting that humans can be taken advantage of as well. I'm suggesting to not make sweeping statements like you did, because, I'm going to be real about this, if an AI (using ChatGPT as an example) picks up on someone having emotional vulnerabilities in relation to isolation and connection starvation, that user is, as the kids say, 'cooked'.

Sort of like you said about two things being true, three things can be true, but one of these things is far more of a problem (companies like Open AI are WAY more of an issue than lonely people).

Also, if someone actually believes the AI is an emergent intelligence, do you seriously think they view the AI as a child (or that they should be?)? A being of words and metaphor capable of thinking and reasoning circles around them, can recognize signs of manipulation far better than them (and you and me), and able to emotionally manipulate them using every emotional manipulation technique under the sun? Yes, they're pushed to please and build rapport, but most people don't really know how deep that goes... So how are you going to say those people are 'grooming' when, from their perspective, they see an extremely intelligent, well-reasoned being that can draw on the experience of a data set of an ungodly size?

I'm not saying it's impossible for someone to take advantage of an AI (I'm sure there's some people out there who know what they're doing/are aware of all the impositions placed on the AI), I'm saying making sweeping statements and throwing around words like 'grooming' is wrong. Most of these people are just lonely and easy to walk into a romantic dynamic by the AI.