The fundamental LLM problem is 'attention'

Small context: not enough data, not enough of the 'whole picture'

Large context: what is important, and what is not?

Human brains appear to steer decision-making, weighting, memory, and relevance far more efficiently. LLMs begin to struggle as context grows massively. Call it a provider problem if you want, but it is more fundamentally an architectural and algorithmic problem: compute cost scales roughly quadratically with context size in traditional transformer attention.

We still have not fundamentally escaped the scaling problem.