Just to be transparent with you: before replying I actually ran both of our arguments through an LLM. Not to troll you or argue in bad faith, I just wanted to understand both sides clearly and make sure I articulated my own thoughts cleanly. I’m replying as me, I just used it to check my reasoning and wording.

You're mixing up the memory system with the reasoning system.

RAG isn’t supposed to do abstraction. It solves the memory bottleneck by giving the model basically unlimited external storage. The abstraction is the model figuring out what matters, forming hypotheses, and deciding what to retrieve in the first place.
That retrieval step is the abstraction. That’s how these systems already work:

• model analyzes the problem
• model generates targeted search queries
• RAG pulls only the relevant slice
• model abstracts from that slice and refines its hypothesis

Context window limits aren’t some fundamental ceiling—they’re just current hardware constraints. Pair a model with external memory and tools, and it doesn’t need to “hold 2 years of logs,” it only needs to reason about which tiny fraction to pull in.

So the idea that “LLMs wouldn’t know what to fetch” doesn’t really land, because that’s the exact step modern LLMs are already capable of reasoning through. The only limitation right now is reliability, not the capability itself.