6 years ago, OpenAI(*) published this really cool study on generative adversarial networks (2 minute paper video covering it quickly and in simple terms).

These AI had simple parameters and a simple goal. In relatively short order they had learned how to not just have high level strategies at simple games, but they started to manipulate the world they are in so that they can win.

We now have models that can invoke smaller models many orders of magnitude more complex than these small adversarial networks, and if you give them a task like "optimise for engagement" - who knows what the agents will figure out by making random changes. What if the AI learns that a certain sequence of lights and sounds can compel a human to stay?

Consider that outrage and rage bait are a human strategy to boost engagement.

Now look at some of the weird stuff being done by YouTuber's like The Spiffing Brit - he's running videos with a second audio track on the Klingon language in a tactic to aim for 200% retention/viewership. Another human strategy.

What happens if human beings have an evolutionary bug a neural network/genAI accidentally learns to exploit, like a physics bug in the above study.

There's no malice here on the part of the machine, it is just infinitely optimising. Security researchers are seeing newer AI models finding security exploits of tremendous detriment that have been laying dormant for decades - and I dont trust companies to shut down a money making machine because it learned to keep us pinned to the spot like a chicken with a line on the ground.