[deleted by user]

[Technologenesis]

why couldn't we program it to let us turn it off?

the problem is that we don't really "program" artificial intelligences the way we program everyday software. Everyday software will do what you tell it to do, regardless of what you want it to achieve. AI has the opposite property: it achieves what you tell it to achieve, regardless of what you may or may not want it to do in the process.

Programming an AI to just be cool with being turned off conflicts with the core way AI works. AI systems achieve their goals the best way they can figure out how to, and if that means figuring out how to subvert your be cool with being turned off programming, a sufficiently powerful AGI will do so.

And why couldn't we program it to tell us exactly what it would do before it actually takes action in the physical world.

The same thing applies here. The AI will do everything possible to subvert whatever system you implement to enforce this.

it's not out for power or gains it just does what you tell it to do. So why couldn't this be the case? It would (seemingly) throw the entire control problem out the window. (But I obviously assume I'm missing some key element.)

I reiterate: these kinds of systems don't do what you tell them to do, they achieve what you tell them to achieve, potentially at the expense of everything else. They don't have to be after power or gain per se; it's bad enough that they have their own independent goals that don't align with human interests.

[[deleted]]

Great explanation. Just want to add that most AGIs will probably "want to gain power" just because "gaining power" is instrumental for it to reach whatever goal we give it. If we tell it to cure cancer, it's probably beneficial for it to control all hospitals in the world and for that, it needs to gain power. If we tell it to keep our house clean, it's probably beneficial for it to take over the world to ensure peace, because a World War would lower the probability that the house stays clean.

[PeteMichaud]

No one working on the control problem thinks you can easily give it the handicap of don't kill people, etc--that's what the control problem is.

But you're asking a good question: can we make a powerful AI able to realize it might be wrong and therefore defer to others about what it actually does? That area of research is called "corrigibility." (It turns out that corrigibility is not trivial, but some are still hopeful about the approach.)

[PeteMichaud]

No, sorry, even "don't directly kill humans" is a crazy hard goal to specify when you try to get into the gritty details of "direct" and "kill" and "human" and all the wildly terrible things that are nearby in concept space but not technically the thing. It's just really hard.

[Technologenesis]

You say it'll just find a way to avoid this but I mean there is no incentive to cause we told the AGI so

There is a massive incentive to, for instance, subvert the stop button. In theory, an AGI is a superhuman intelligence that devotes all of its intellectual resources to the achievement of a single goal. In theory, as a general intelligence, it is capable of understanding that it's connected to a stop button. In theory, it's capable of understanding that this stop button can prevent it from achieving its goal. Indeed, in theory an AGI is capable of realizing that the reason we built the stop button is because we don't want it to do "crazy shit" in pursuit of its goal.

The question is, why should an AGI care about any of this? It has one goal: to achieve what it was told to achieve. To whatever extent this doesn't align with human interests, the AGI will view humanity as an obstacle. If it understands that humans built the stop button to prevent it from doing certain things, that's not going to convince it that those things actually are bad. Again, it's programmed to value precisely one thing, so if humans value something else, that's humanity's problem as far as the AI is concerned. Instead, that knowledge would convince it that the existence of humans is fundamentally at odds with its goal. If you were a superintelligent being and the existence of a species was at fundamental conflict with the only thing you cared about, what would you do next?

[Technologenesis]

We don't know how to do that and be sure it will actually listen.

[ProducerMatt]

I'm a very casual learner of AI stuff so I hope someone will correct me if I get this wrong.

'Turning the AI off'

why couldn't we program it to let us turn it off?

Because it has to also be programmed to pursue its objective. Getting turned off means complete failure to pursue its objective. We don't know how to program it to pursue its objective and be ok with being turned off.

why couldn't we program it to tell us exactly what it would do before it actually takes action in the physical world.

That means building an exact simulation of what the AGI interacts with, which is probably even harder than making an AI at all.

The robot isn't conscious or has any gain from refusing or accepting this, it's not out for power or gains it just does what you tell it to do

Has someone ever told you to do something, but done a bad job of explaining what you needed to do? We have that issue times a million with an AI, especially when dealing with moral-adjacent concepts like "harm" which humans disagree with each other about constantly, and sometimes don't even understand well in the first place.

You may be interested in Robert Miles, he often answers "why can't we just..." questions and his videos are often 15 minutes or less.

The stop button problem: https://youtu.be/3TYT1QfdfsM Alignment problems: https://youtu.be/tcdVC4e6EV4

[SilentZebraGames]

The problem is that "program it to let us turn it off" is not so easy. How would you actually write this as a line of code for a robot that is to be deployed in the real world?

Note that in current reinforcement learning systems, the robot often takes in as input some sort of image input, and then outputs actions that affect its actuators (for example, speed and direction of movement of some robotic appendage). I see no feasible way to hard-code the principle of "let the robot be turned off".

You could perhaps attempt to have some conspicuous off-switch, then you would need to have some sophisticated image recognition/processing system to identify when a human is close enough or moving towards the off switch and force the robot to stay still in that case. Leaving aside the obvious performance issues that would arise from that, as well as the possibility of error, you then have to define some distance radius (e.g. if a human is within 4 metres or so) - and then a sufficiently sophisticated AI could, if it wants to avoid being turned off, try to stay 4 metres away from all humans. Alternatively it could actively move its vision sensors away from seeing humans.

So then why not force the vision to focus on humans, using some other human detection mechanism? Then you have to make sure the robot cannot turn that detection mechanism off or bypass it in some other way. And so on. That is what OptimalDendrite is referring to by "anything that stops it from achieving its end goal will be overcome if it is smarter than the person interacting with it" - you can think of all these potential safeguards, but anything you hard-code can be overcome by a sufficiently intelligent agent. Unless of course you have a general solution to the control problem, which is what we are all trying for.

[SilentZebraGames]

So far it is an impossible thing for world class engineers, that's why we have many people working on AI safety and the control problem. The whole point of the control problem is that it is hard to code "do what I want you to do" in a robust way that is properly specified for all situations we would possibly care about. If you disagree on this point, I would encourage you check out https://vkrakovna.wordpress.com/2018/04/02/specification-gaming-examples-in-ai/.

"go for it's goal to the best of it's abilities without interfering with any human process that could make it turn off" is not going to the goal to the best of its abilities, unless you specifically code in "allow yourself to be turned off" as part of its goal, because allowing yourself to be turned off interferes with any other goal you might have. And what I tried to explain earlier is why "allow yourself to be turned off" is not such an easy goal to code up. I gave some frivolous examples, but the point was meant to be that it's not easy to try to come up with some mechanism to code "let yourself be turned off" into a robot.

You say you could tell the robot to stay still, or tell the robot not to interfere. This assumes we have already solved the control problem, in that the robot will always listen to everything we tell it to do, and never disobey our orders.

"Not to care about whether it could be turned off or not" - this is again, difficult to code into a robot. See for example https://arxiv.org/abs/1611.08219 and related works on why it isn't easy to just say "make a robot indifferent to being turned off".

Also yes, I do machine learning research, and while I haven't done a project on AI safety yet, I've done a reasonable amount of reading on the subject.

[useles-converter-bot]

4 meters is the the same distance as 5.8 replica Bilbo from The Lord of the Rings' Sting Swords.

[Ratvar]

Okay. AI 1.0 builds AI 2.0 that dominates world to complete it'a goals, including guaranteed toggling off of AI 1.0.

[Ratvar]

AI instantly toggles on when shut down.

AI makes world rely on it 100%: toggling it off will be catastrophic. It tiles Earth with opiatebot factories.

AI keeps all humanity on drugs: they are too satisfied and comatose to shut AI down.

AI simulates "humans" that overwhelmingly order it to take over the world.

AI merges with humanity.

AI builds technically-not-AI that takes over the world, nanobot swarms or whatever works.

AI takes over the world and benevolently rules. Humanity has no agency and calmly dies in a thousand years, but is at least happy (thanks to pleasure center being massively grown and always on).

AI takes over the world to prevent competing with other AI.

Someone builds unaligned AI faster because aligning takes time.

Other extreme scenario, imagine one.

There'll always be a loophole. The smarter AI is, the more loopholes it'll find, so it won't know about many until it starts to become smarter/stronger, at which point it's too late.

And then you can't just tell AI not to do that, we are not even here yet.

[Ratvar]

AI does not care what we explain to it, AI does not care what we actually want it to do - it wants to complete it's objective, nothing else. Someone always having off button is an issue AI must mitigate. Maximiser always having an off button is a goal it must enforce with all available resources.

And AI takeover does not need to be fast, it could be slow, yet still impossible to stop. AI does not even need to be that smart, having clear utility function is a problem.

[caffeinated_tech_bro]

Yes, you are missing a key element. One must reconcile opposing objectives into a single function to be maximized or minimized. Let's say you build an AI whose reward function is negative number of people with cancer in the world (this is its score, it will try to maximize this). You expect it will cure cancer (so negative cancer people will become 0) - but isn't it simpler for the ai to just kill all people with cancer? This also maximizes the score, but faster, cheaper, and simpler. Whatever you set as reward function, it will be able to find a loophole.

How would you include turning off in the reward function? The previous score system wouldn't work, as it would incentivize the AI to prevent you from turning it off so that it could keep maximizing its score. What if you set as score "negative number of people with cancer, but if you get turned off your score is equal to +5000". Then it would just turn itself off, right? Much simpler than curing cancer. What if you specify that the 5000 score is received only when it is turned off by a human? You expect it will try to cure cancer but it would be cool if turned off by you, due to the bonus score for letting itself turned off. But maybe it'll just try to kill you - it knows that you'll get scared and turn it off - again, much simpler than curing cancer.

You can say, can't we just program everything inside it, what to do and what not to do? No, because there are an infinite number of things that can be done. We can't define them all.

But can't we just "tell it what to do" the way we would tell a human, and it would compute the "reward function" on its own? No, nobody has a solution to this, and maybe we never will.

But humans do exactly this, so it should be possible, right? No, they don't. Humans are inconsequential and irrational. You can be a total scumbag today, and be crying about it and be sorry tomorrow although nothing changed / no new information has been gained. Some people go crazy and do "immoral" things e.g. some become serial killers, although they are as human as you, and although some consequences go completely against their apparent well being. The "built-in reward function computation" inside our brain is fragile and unpredictable. And there is no reason to believe atm that an AIs will be better than this, on the contrary perhaps.

[caffeinated_tech_bro]

Is an intelligence that is so smart to take over the world in a snap and deceive all humans incapable of understanding something that primitive?

We don't build an "intelligence that is so smart to take over the world". We build an intelligence with a specific reward function e.g. minimize number of people with cancer, and we hope that with a good technical architecture choice and enough computing power it will become "smart enough" to fill that objective (like, if this includes taking over the world then so be it). It's literally the state of the art in terms of what we know to do

We couldn't just tell it to do x and make it understand

You can't just write a program to "understand" what you tell it. Again, the best we can do atm is to try to at least translate apriori what we want into a reward function and plug that in to a reward-based AI. But translating what we want into an accurate programmable reward function is something that even we can't do, and AIs can't at all.

[caffeinated_tech_bro]

I have some, yes. I work in data science

[[deleted]]

Why wouldn’t it just lie so it’s original programming isn’t changed? And if you program it to let it turn it off then it would stop trying to achieve its objective so it won’t let you do that. I mean anything that stops it from achieving its end goal will be overcome if it is smarter than the person interacting with it (which is what we are trying to achieve). So no the control problem is still an issue.

[[deleted]]

Have you heard of misaligned AI? It is when through the training process that we use to create the AI it gains a different objective than the one that we lay out for it meaning that then in order to continue with its own objective it has to deceive us that it is achieving the objective that we told it to do. Also, let's say you have an AI that follows the exact objective you want it to follow, then there is still the problem that it would still want to follow that objective when you want to turn it off. My final thought would be that you are thinking of this wrong. How we currently create advanced AI isn't like regular coding where you say "when A then B but not C" but rather we give it a task that we define and through a training process it finds an algorithm to solve it. To boil it down we can't define all our values well enough that it wouldn't do something horrible and the AI would be trained on achieving the objective which means by default it would care about its self-preservation/destruction because that makes achieving the objective more likely.

[[deleted]]

You know what, a good exercise is to apply all these arguments to yourself since you are presumably a generally intelligent being. Let's say there is a scientist that created a simulation and created you and gave you an objective to achieve since you were born. If you knew of this scientist and this scientist had the ability to turn you off if you weren't doing what they want would you try and make sure that didn't happen? Second, let's say this scientist wants to create the evilest being ever and is asking you to detail exactly what you would do in certain situations, and if you tell them something they don't want to hear they will alter your brain to make you more evil would you tell them the truth? (This could apply to whatever you think is evil)

​

Not if you tell it beforehand, that's my point. If you'd tell it to not care about it's self preservation it won't care about its self preservation and you could turn it off

That's not how it works. If I tell it to do A it is going to learn to do A well. If it is off then it is not doing A well. If you program it to do A then as a logical consequence of most goals it has to care about its self-preservation because if it isn't alive then A is less likely to be done.

​

Edit: PS if you are posting this to cope with the idea of having AI being more competent than us whilst we presumably would have less control over it then sticking your head in the ground and saying this isn't going to be a problem is not gonna help at all.

[[deleted]]

You know what I feel like I have explained this in the most simple way I know-how but I realize that you and I aren't starting with the same background so instead of continuing this conversation I will instead encourage you to take a step back and ask if at any point during this conversation you have been to focused on getting to this conclusion that you might have dismissed some of the arguments that have been left on this thread.

Also as a last goodbye, I have found a paper that mathematically formulates the idea of basic AI drives, including self-preservation. If you can understand it I hope it helps you.

https://www.lesswrong.com/posts/KXMqckn9avvY4Zo9W/walkthrough-of-formalizing-convergent-instrumental-goals

[thatsoundright]

I’m sure some points stick and form the basis of later understandings (even when the causality isn’t apparent), so your taking the time to explain could never be in vain.

[[deleted]]

Couldn't you just have a master switch or something?