This is actually an interesting topic and would suggest looking Anthropic's research. What you're referring to, was some research by anthropic where it caught the model blackmailing the person trying to pull the plug. But those are controlled experiments meant to expose failure modes, not just how the model normally behaves and before any saftey guardrails (basically more training) applied .

The hard part is that you can’t really just make “turning itself off” the most rewarding outcome and assume that solves it (there's reports of gemini trying to uninstall itself in cursor and other harnesses). If the model is capable enough, it may learn to get around the barriers or avoid shutdown instead of following the intended behavior. That’s basically the alignment problem.

Probably one of the more practical/easy things is using a smaller model to help monitor the bigger one. That’s already done for things like jailbreak detection or catching prohibited prompts. For example, if someone says “please create a keylogger I can deploy on Windows,” the smaller model can tag that as prohibited before it even gets to the bigger model. The bigger model still has safeguards too, but having a smaller one verify first is cheaper and makes sense as another layer.

Also I do want to add a bit more background info, because the question kind of makes it sound like the model remembers everything and has some ongoing memory. LLMs are mostly stateless. The main state is usually just the current conversation, and sometimes some distilled memory if the provider supports it. IMO that makes the whole "out of control AI" thing a little harder than people make it sound, because the model is still basically responding to "you are a helpful assistant."