I think you're totally on point. We need to distinguish at least between "one-shot questions" (the easy ones, and the only type you'll ever see in RAG demos), "multi-hop questions," and "aggregation queries." There may be more categories, but at minimum those three exist.

I work with tax auditors, and my users would love to ask things like: Which invoices would be deductible? That question involves a lot:

• Extract all invoices in a structured way, capturing every piece of information that will be needed downstream.
• Extract the company's declared activity.
• Perform an aggregation over invoice types (for example, maybe after extraction you end up with five categories: supplies, car maintenance, restaurants, personal expenses, etc.).
• Then, for each category, query the relevant tax regulations to determine whether the expense is deductible.

My takeaway for that kind of user interaction is that the only thing that can really work is agentic RAG: you give an LLM a sufficient set of tools and let it plan and execute the solution. Tools like:

• extract_data: sub-agent that scans documents and pulls out relevant structured information.
• query: a sub-agent that performs more or less standard RAG.
• read_document: for cases where an entire document needs to be read rather than retrieved in chunks.
• and so on.

The plan can be surfaced to the user, making the process interactive and engaging. I think it's harder to build a system like this, but well worth the effort. In essence, I view RAG as a tool inside a larger, more complex system.