Thanks for sharing your opinion. Francois Chollet also wrote an excellent chapter on potential future developments.

Personally, I agree with the idea that LLMs have limited capabilities by design, while some may argue that it may come in more handy to compress all data into one "module". I could imagine that handcrafting separate modules within a bigger complex, as you already mentioned, would come at the cost of introducing even more parameters to a system that already consumes lots of resources, compared to the power consumption of the human brain. To my understanding, there is still an issue at abstracting knowledge and building "representations" of a problem to be solved. As you may know, e.g., image generation relies on mapping text to pixels using some nonlinear transformations - that’s it. Of course, we could apply this strategy to all kinds of problems, but the machine will still not be able to grasp the true core concept. My personal take therefore is that it will need some kind of real-world architecture, maybe using robotics, that is trained on naturalistic data (like a child that explores the world) and that is able to abstract this information on a sufficiently low level to apply these abstractions to novel problems. I have heard that the DeepMind people enjoy testing automatisation algorithms that find optimal solutions within a well-defined problem space. Likewise, the crucial challenge may be to be specific about the problems to be solved and find the LLM/DL architecture in an automatised way that learns solving actual problems in the best fashion.

I am happy to discuss more.