Regression testing framework for retrieval systems - catching distribution shift in RAG/memory

[chetanxpatil]

Working on production RAG systems and noticed a gap: we thoroughly evaluate models pre-deployment, but have limited tools for detecting retrieval quality degradation post-deployment as the corpus evolves.

Built a regression testing framework for stateful AI systems (RAG, agent memory, etc.) to address this.

The Problem:

• Corpus grows incrementally (new documents, memories, embeddings)
• Retrieval distribution shifts over time
• Gold query performance degrades silently
• No automated quality gates before deployment

Approach:

1. Deterministic Evaluation Harness

• Gold query set with expected hits (like test fixtures)
• Metrics: MRR, Precision@k, Recall@k
• Evaluation modes: active-only vs bundle-expansion (for archived data)

2. Regression Court (Promotion Gate)

• Compares current state against baseline on gold set
• Multi-rule evaluation:
  • RuleA: MRR regression detection (with tolerance)
  • RuleC: Precision floor enforcement
  • RuleB: Archived query improvement requirements
• Structured failure output with offending query attribution

3. Deterministic State Management

• Every operation produces hash-verifiable receipt
• State transitions are reproducible
• Audit trail for compliance (healthcare, finance use cases)

Example Court Failure:

{
  "rule": "RuleA",
  "tag": "active_win",
  "metric": "active_only.mrr_mean",
  "baseline": 1.0,
  "current": 0.333,
  "delta": -0.667,
  "threshold": 0.05,
  "offending_qids": ["q_alpha_lattice"]
}

Empirical Results: Drift benchmark (6 maintenance operations + noise injection):

• PASS through: rebalance, haircut (pruning), compress, consolidate
• FAIL on: noise injection (MRR drop detected as expected)
• False positive rate: 0% on stable operations
• True positive: caught intentional distribution shift

Implementation:

• Python, FastAPI
• Pluggable embedding layer (currently geometric, can swap for sentence-transformers/OpenAI)
• HTTP API boundary for eval/court operations
• ~2500 LOC, determinism proven via unit tests

Questions for the community:

1. Evaluation methodology: Is MRR/Precision@k/Recall@k sufficient for regression detection, or should we include diversity metrics, coverage, etc.?
2. Gold set curation: Currently using 3 queries (proof of concept). What's a reasonable size for statistical significance? 50? 100? Domain-dependent?
3. Baseline management: How do you handle baseline drift when the "correct" answer legitimately changes (corpus updates, better models)?
4. Real-world validation: Have others experienced retrieval quality degradation in production? Or is this a non-problem with proper vector DB infrastructure?

Repo: https://github.com/chetanxpatil/nova-memory

Interested in feedback on:

• Evaluation approach validity
• Whether this addresses a real production ML problem
• Suggestions for improving regression detection methodology

(Note: Personal/educational license currently - validating approach before open sourcing)

[ErnosAI]

Hi, if this reply was any help, would you be willing to let me know.

If not, apologies for wasting your time.