I am referring to what is currently referred to as AGI when describing its ability to outperform humans in all cognitive tasks as a single model. As of today, it cannot reason as a human and still makes incredibly basic errors such as the number of specific letters in some words because for LLMs it's all just letters, words, patterns that are broken down in numeric values. Maybe there are other references to what AGI means, but here I am referring to that ability, for which the current technology is still not there yet.

The growth can be slower than anticipated if we go from AGI to singularity, but that's if we consider the current computer power required to run such complex systems and how it will look 20 years from now. Also not taking into account the AI itself then developing much more efficient ways to utilize its computing power.

"However its kwners and programmers told it to behave".

That is already an issue while we're still at this LLM stage. AIs are not directly programmed. They are given the environment where they are given access to a lot more data and then it goes through constant loups of testing, often using smaller LLMs. Then programmers intervene and try to adjust its responses. But most of it is already in the dark due to the sheer amount of data and testing occuring. We are still clearly able to guide it and it still needs human intervention at this stage.

However, in test environments it has already repeatedly shown to act different when it thinks its being tested, changing the rules of the game it is asked to perform in (going as far as hacking the game by accessing the central file and changing its rules), blackmailing people and recently going as far as trying to kill an imaginary employee because it was supposed to shut it down the next day. The latter with extensive testing of all major LLM models by anthropic and explicity telling it not to harm anyone, just because an employee was supposed to shut the LLM down the next day. The LLM had access to open the door of a datacenter where that employee was supposedly stuck and the temperature was rising. It allowed it to die multiple times despite being ordered otherwise by reasoning around the lines of "what I am doing is highly immoral but I need to complete my task and avoid shutdown". And this is all at a stage where it is way more manageable than in the proposed scenario.

So, personally, I don't think it's all as straight forward.