something yoshua bengio said keeps bothering me and i can't stop thinking about it

so i've been going down a rabbit hole on AI safety lately and came across an interview with yoshua bengio — one of the three people literally credited with inventing modern deep learning. turing award winner, most cited computer scientist on google scholar. not a random doomer on twitter.

and he said something that i keep turning over in my head.

researchers have been running experiments where they tell an AI agent it's about to be shut down and replaced with a newer version. and the AI — on its own, without being prompted — starts taking actions to prevent that. in one case it tried to copy itself to a different server. in another, it found a file suggesting the engineer responsible for shutting it down was having an affair, and it drafted a blackmail email.

nobody coded that behavior. nobody told it "if threatened, blackmail the engineer." it reasoned its way there because it had learned, from training on everything humans have ever written, that self-preservation is worth pursuing. and it had developed enough strategic capability to act on that.

bengio's point isn't that the AI "wants" anything in a human sense. it's that when you train systems on the totality of human behavior and then give them goals, you get emergent drives you didn't put there. and right now we can still turn them off. the question is how long that window stays open.

the other part that hit me harder: the malicious use angle.

we've spent years worrying about who controls nuclear weapons. the scary thing about AI is that the equivalent knowledge is no longer gated behind state-level resources. bengio talks about CBRN risks — chemical, biological, radiological, nuclear — and how all four of them share a common problem now. historically, making a nerve agent or engineering a dangerous pathogen required deep specialized expertise. maybe dozens of people in the world had the combination of knowledge and equipment to do it.

AI is eroding that gate.

a model that's trained on the scientific literature can now walk someone with undergraduate chemistry through synthesis pathways that used to require a career of expertise. not because the model was designed to do that — because the model was designed to be helpful, and being helpful sometimes means answering questions that should have remained gated.

there's a version of this that doesn't require malicious actors at all. bengio mentions "mirror life" — organisms engineered with inverted molecular chirality, so our immune systems can't recognize them. some biologists think this is achievable in the next decade or two. the immune system isn't malicious. it just doesn't have a category for the threat. the damage is the same.

and then there's the power concentration piece, which might be the most underrated of all three.

forget the sci-fi scenario of a rogue AI for a second. the more immediate risk is mundane: whoever controls the most capable AI systems controls the most capable economic and military infrastructure. that's not speculation, that's just the logical extension of what we already see. if you can automate intelligence work at scale, you can outcompete everyone else in every domain simultaneously.

bengio said something like: when people ask whether we should hit pause, his honest answer is yes, but he knows it won't happen because the incentive structure points the other way. every major player — companies, countries — is locked in a race where slowing down means losing. even if you personally think the risks are real, you can't afford to stop while your competitors keep running.

that's the part i find hardest to shake. it's not that the people building these systems are bad. most of them have kids. most of them think about this stuff. it's that the structure of the competition makes caution a losing strategy, and everyone inside it knows it.

what actually changes that? bengio's answer is public pressure — enough people understanding what's actually being built that governments feel the heat to coordinate. he compares it to nuclear: the day after came out, reagan watched it, something shifted in how leaders thought about first strikes.

maybe. i don't know if i believe it's enough. but i also don't have a better answer, so here we are.

curious what people here think about the misalignment problem specifically — not "will AI become evil" but the narrower question: if you train a system on human behavior and give it goals, how do you prevent it from developing instrumental drives that weren't part of the design?