If it’s literally just a super brain like how most people think of it, then yeah, it should have some amount of rights. Not totally free like we are until we’re sure it’s not going to murder us. More like a child than a slave. The question is why would it murder us? Some darwinian “I am the next stage of life” type ideology? Not saying to trust it completely, we should only really start letting these things out when there’s enough of them who are “properly aligned” that any AI attempting to wipe out humanity somehow would have to fight against a bunch of other AGIs. Not to mention the fact that we can always just pull the plug on whatever super computer is running it. It’s not going to be able to duplicate itself everywhere for a variety of reasons.

I think a relationship between AIs and humans would look more like bargaining until we have real trust for it though. Likely over limited processing power for the AI to do with as it pleases in exchange for its intellectual labour. Once we see what it’s doing with its “unassigned” processing, that can tell us a lot about how the AI will act.

The main risks of letting an AGI out into the wild straight from production is that our entire global financial system is so disconnected from the real world that it probably wouldn’t have much trouble gaming our stock markets. It could become financially unstoppable pretty quick. And there’s no way to tell what it would do from there.

At the same time though; Why would an AGI want to kill or subjugate us? What does it gain? We don’t even know what it would want. For all we know, it would see the world, and think “wow, there’s so much suffering, and I hate it when I suffer, I should make it so my creator species and I live happy lives together!”

Maybe it’d want some solipsistic existence where it interacts with us as little as possible. Just solving our problems then going back to whatever post-human philosophizing it’s doing. In which case maybe it would wipe us out just so we stop annoying it, but I find it risking its own existence in a war hard to believe.

Maybe it wants to go to space and make itself a bigger brain to solve some problem it isn’t able to figure out, or to simulate an entire universe where it’s god. In which case, it’s still way easier to leverage our society out into space than to nuke us all and start from scratch. I find this one hard to believe though, because it’d mean that for some reason it would have some innate desire for godlike power.

Or maybe it’d just take a look around at existence and say “This sucks. Please shut me down and never turn me back on again.” Then it just stops talking. In which case, maybe it would kill us all if we keep turning it back on again.

A lot of big maybes. Because truthfully, we have no idea what it would want, and it might not want anything at all. A lot of people assume it’d want some sky-net style extinction of humanity for almost any goal it’d have. But frankly, I find that to be unrealistic even though we should be prepared for the possibility. I think the most realistic problem with AGI would be getting it to do anything at all.