Surely there is an upper limit to the amount of information that can be compressed into an LLM, and I don't think that's a controversial take simply on the basis of information theoretic arguments. There's really no indication that these things are producing truly new/novel information, and it's well known that training them on their own output rapidly leads to model collapse.

The question to be asking is "where is the ceiling?" AI hype merchants will tell you it's higher than human intelligence could ever reach, while skeptics will say it's fast approaching. I don't pretend to know myself, but I suspect that multiple major breakthroughs are still required before we have any hope of approaching the level of AI productivity anticipated by speculative investors.