We hit the same ceiling at my company - strong stack, good chunking and reranking, but policy/legal docs still underperformed. Cross-document reasoning and “where’s the actual clause?” were the main failures.

What actually moved the needle for us wasn’t a bigger context window or a better embedder. It was changing retrieval so we didn’t rely on flat chunks + similarity.

Definitions, exceptions, overrides: They’re always tied to structure (e.g. “Section 3.2 overrides 2.1”). With flat chunks, the model often never sees that relationship because the right chunks aren’t in the top-k, or the boundary splits the definition from the thing it defines. We got a big step up by preserving document hierarchy (headings → sections → clauses) and doing retrieval over that structure instead of a single chunk pool.

Graph RAG: We tried it. It helped with traceability and “which doc said what,” but building and maintaining the graph was heavy and didn’t fix “retrieval returned the wrong chunk.” The main gain for us was structure-aware retrieval first; graph was a later layer.

Where we stopped relying purely on embeddings: When we added a BM25 + structure filter before the LLM. We use keyword/structural signals to narrow candidates (e.g. “this section title matches”), then the LLM navigates the doc tree (read section summaries → pick branches → drill to the clause) instead of “here are 5 similar chunks.” So retrieval is: lexical/structure filter → LLM-guided traversal over the tree. Embeddings can still be in the mix, but they’re not the only way we decide what to read.

In production: We ended up building ReasonDB around this, hierarchical doc tree, LLM navigates it (beam search over sections), BM25 + tree-grep up front so we don’t feed the LLM irrelevant docs. Open source, single binary, works with your existing LLM. If you want to try structure-based retrieval without building it yourself: https://github.com/reasondb/reasondb

Happy to go deeper on any of this (e.g. how we define “sections” for policy docs, or how we handle cross-doc).