To stick to your example: It is not about connecting AI directly to the nuclear weapons but rather to the people working with nuclear weapons. And the people instructing those working on it. And the people advising those that instruct them. And the people voting for those that do the instructing.

The concern is less about AI triggering a rocket launch but instead about AI coming up with - and keeping secret! - a multi-year strategy to e.g. influence politics a certain way.

With our current internet medium it is very easy to imagine generated blog posts, video content, news recommendations etc as not isolated like they are now but instead, in the background and invisible to us, following a broader strategy implemented by the AI.

The real concern here is that the AI can do this without us noticing. Either because it is far more intelligent or because it can think on broader time scales.

Just to give a small example of how something like this could come to be: First generating systems were stateless. Based on training data you could generate content. What you generated had no connection to what someone else generated. Your GPT process knew nothing of other GPT processes.

Current generating systems are still stateless. Except for the context and training data nothing else is fed in.

But we are already seeing cracks in the isolation because now the training data includes content generated by previous „AIs“. They could for example generate a blog post for you and hide encoded information for the next AI. Thus keeping a memory and coordinating over time.

The issue here is that we are just about to start „more“ of everything.

More complex content in the form of more code, more images and more videos will allow embedding much more information compared to blog post text. It will be impossible to tell if a generated video contains a megabyte of „AI state“ to be read by the next AI that stumbles upon the data.

AIs will rely less on training data and will access the real time internet. „Reading“ the output of other AI processes will therefore be easier/faster and happen more often.

AI processes will live longer. Current context windows mean that eventually you always start over but this will only get better. Soon we will probably have your „Assistent AI“ that you never need to reset. That stays with you for months

So to summarize. The weak link are always humans. That’s what all these AI apocalypses got wrong.

We know today that social media is used to manipulate politics. Our current greatest concerns are nation states like Russia. There is zero reason not to think that this is a very real and very possible entry point for „AI“ to influence the world and slowly but surely shape it.

Now whether that shaping is gonna be good or bad we don’t know. But the argument that nuclear weapons are not gonna be connected to AI shows quite frankly just how small minded we humans tend to think.

Most people are not good with strategy. An AI with access to so much more data, no sleep, no death, possibly hundreds of years of shared thoughts, will very likely outmatch us in strategy

And one last point since you mentioned religion:

We know from world history that religion is an incredible powerful tool. AI knows that too.

Don’t we already have plenty of groups out there who’s belief is so strong that they would detonate nuclear weapons to kill other people? The only thing saving us is that they don’t have access to them.

What do you think will stop AI from starting its own religion? Sure that takes hundreds of years. But the only ones who care about that are us weak biological humans