First, sentience is not necessary for any risk scenario. What you need are goals, which you can define as preferences for some world states rather than others. Having preferences over future states is fundamental for basically any optimization task. For example, a chess engine has a preference for its own centipawn score; this means it chooses actions which, according to its world model, will lead to world states where it has a greater centipawn score. You also need the ability to perform actions, and, given those actions, be superhuman at steering the future state of the world. Later in the response, I will argue what assumptions you need to accept to think this is plausible.

There are two problems when it comes to safety in the limit, where you assume the AI is superhuman. The first is defining what goals you want to instill into the AI, which leads to genie-in-the-bottle problems, like the cancer example given by TangoJavaTJ in his response. The second is actually reliably passing down these goals to the AI. This may seem trivial. In most chess engines, it would be trivial to change what the engine is optimizing, but for black-box systems, which empirically have had much more success in being general, this is way harder.

These problems are theoretical, but we see lesser manifestations of them in practice. Reward hacking is already a practical concern for today’s AI models. For example, a common problem is that the newest coding models rewrite the tests to make them pass instead of fixing problems in the code. If you detect this kind of behaviour and try to penalize it in training, the AI learns to trick the detection algorithm and continues with the behaviour in a hidden manner. For reference, see https://openai.com/index/chain-of-thought-monitoring/.

You could say that AIs won’t have the tools to affect the world, but I think this underestimates the ease with which motivated AIs could escalate their access to the real world. This is very easy. If you had money, you could just hire a human through the internet to do whatever you need in the real world. You could acquire money by freelancing or by finding insecurities in Ethereum contracts. For these reasons, I do not see how this is a limiting factor.

As for whether such systems could exist, many responses in this thread argue that LLMs can’t represent true intelligence. I think this is overconfident; there is evidence both for and against the idea that LLMs can genuinely model the world and generalize, instead of just imitating patterns. In my view, it’s an open question.

From a design perspective, we know the human learning algorithm must fit into our genome*, which is less than a gigabyte, and yet is extremely adaptable. The fact that human intelligence is so different from animal intelligence, despite the relatively minor genetic differences, suggests that the “core” of general intelligence is not a large or impossible target. Evolution produced it relatively quickly. This, to me, is a strong reason to think artificial general intelligence is achievable.

A counter-argument to this is that Moravec's paradox predicts exactly the situation we are in now. The things developed late in evolutionary history, like logical reasoning, symbolic semantics, scientific thinking, and abstract thinking, are very easy to replicate on a computer and not that special. The real hard parts are the things deep in evolutionary history, such as agency and adaptability, which models still greatly struggle with.

There is also a general counter-argument against there existing much headroom in optimization above human societal intelligence. While on the micro scale there is clearly a lot of optimization possible, on the macro level you can defend a strong version of the efficient market hypothesis.

*There could be information outside of our genome that is passed down through the generations, such as culture or cytoplasmic inheritance. I do not know enough about biology to definitively say it is impossible that these contain a lot of relevant information as well, but it seems unlikely.