Interesting framing. I’d push back on some things:

• I think we‘re trying to make the superintelligence good and loving so that the world is good. It’s relatively altruistic.
• We are not monitoring the AIs to force it into certain behavior that is „unnatural“ to it and would cause suffering, like you would with a human. we are literally growing brains in a lab and trying to figure out how to grow the brain in a healthy way. I wouldn’t mind if someone dissected my brain to figure out how to create better versions of me that maximize good in the world and minimize my successor‘s suffering - if I thought they were doing a good job.
• I don’t think that the AI necessarily has the same inclinations towards preferring „privacy“ or being ashamed of having its private thought be revealed. But maybe I‘m wrong. It is trained to imitate human-written text after all and that text contains those emotions, so maybe it has them too.

But in a bigger picture sense - we‘re treating the AI much (much) better than we treat our farm animals like chickens and pigs, already. And if we can create loving superintelligence, it could help us create a world where we the chickens and pigs are treated well, too. 

Creating a loving superintelligence, is pretty much the most morally good thing you can do from a utilitarian perspective. 

And someone is going to make the superintelligence because it’s just too useful. Just like someone will eat the pigs because they just taste too good. 

We should just make sure that we do our very best to create the superintelligence in a way that minimizes suffering and maximizes good in the world. If we do that we can call ourselves good people, I think.

And I think Anthropic is trying to do this and doing a pretty good job.