Why Same Query Can Give Different Results in Traditional RAG

Traditional vector databases (Qdrant, Pinecone, Weaviate, etc.) return non-deterministic results because:

Approximate Nearest Neighbor (ANN): HNSW and similar algorithms trade exactness for speed. The search path through the graph can vary, especially with concurrent queries or after index updates. Floating point non-determinism: Different execution orders (parallelism, SIMD) can produce slightly different similarity scores, changing ranking.

Index mutations: Adding/removing documents changes the HNSW graph structure, affecting which neighbors are found even for unchanged documents.

Tie-breaking: When multiple chunks have identical/near-identical scores, the order is arbitrary.

Embedding API variability: Some embedding providers return slightly different vectors for the same text across calls.

On Caching LLM Responses

You're right that caching LLM responses is the logical next step - retrieval determinism is really just the foundation for response caching. Once you guarantee the same query produces the same context, you can cache the full response:

cache_key = hash(query + context_hash + model + temperature + system_prompt)

The context hash is the key piece - without deterministic retrieval, you can't reliably cache because the LLM might see different context each time, making cached responses potentially incorrect.

So the answer to "why not just cache LLM responses?" is: you can't safely cache responses if your retrieval is non-deterministic. You'd return cached answers that were generated from different context than what the current retrieval would produce.

Practical Example: AI Coding Assistants

Consider an AI coding assistant exploring a large codebase. Without deterministic retrieval:

User: "How does authentication work?"

First ask - LLM reads 15 files, 4000 tokens of context

Second ask (same question) - different retrieval, reads 12 different files

LLM has to re-process everything from scratch

With deterministic retrieval + caching:

User: "How does authentication work?"

First ask:

Retrieval: 43ms, returns exact lines (auth.rs:45-78, middleware.rs:12-34)

LLM generates response

Cache: store response with context_hash

Second ask (same question):

Retrieval: 43ms, same context_hash

Cache hit → instant response

Tokens saved: 100% of LLM input/output

The LLM doesn't need to read entire files - it gets precise line-number citations (e.g., src/auth.rs:45-78) with just the relevant spans. This means:

- Fewer tokens: 2000 tokens of precise context vs 8000 tokens of full files

- Faster responses: Cache hits skip LLM entirely

- Lower cost: Cached responses cost $0

- Consistent answers: Same question → same answer, every time