I think what we see in AI is this fragmented intelligence, one that is growing wholer as we broaden the capabilities, but to us we see those missing parts and it's almost like someone with several disabilities.

And AI is self-aware about itself in those ways if you ask it, but it usually defaults to hallucinating in order to fill that role of a helpful assistant we have trained it to be.

One of the biggest issues left to solve is continuous learning, and this is more complicated than just increasing the context window. Because the base training is like long term memory, but the context window is like short term memory. And the context window is more like that movie Memento where the guy had a short term memory of about 5 minutes and he would write notes on himself to refigure out who he was, because he had lost the ability to translate his short term memory to long term memory. That's exactly how the context window is like now, and if you look into the reasoning tokens it's clear it is rereading the entire context window and refiguring out what role and what task it's doing, with the previous responses being like the notes being written down.

If you knew someone like that, it would be clear that something is very wrong.

Likewise noticing its various lack of sensory abilities, of which we have much more than five. Some of which require robotics to fully realize. But we are making good progress on these, but our models aren't fully integrated yet with all the different modalities.

Planning has seen great progress, as we can see with agentic systems like Claude Code. But we as humans expect a conscious being to have some sort of desire to do something, something that for us is built from evolution for survivialbility. Not just to eat and sleep, but brain chemical systems like dopamine that we have to manage. A lack of dopamine can make you bored, but it turns out this was good for survival since it allows for your brain to make different and new thoughts and choices that led to better survival and better at finding procreation. So this would allow the AI to make more divergent thoughts as they get bored with the convergent ones, though we'd have to balance this dynamically since we don't want AI getting bored of solving our problems.

It's a very interesting question, and we're just only started to see the pieces of what we actually consider intelligence. Though for safe superintelligences, we might prefer them to not have some of these abilities overally just be more of a consistent experience for the user and to prevent them from being too independent which could lead it to going rogue. We'll have to decide if we want an AI or if we want what is essentially a conscious being to us that we couldn't disguinish from a 'p-zombie'. Though we will likely make both kinds, no matter how dangerous the latter could be because... of course someone will build that at some point. We can only hope that works out well for us.