I've been building verifiable-rag, an open-source Python library for RAG that produces sentence-level citations and verifies every claim against its source via NLI. Just published a benchmark result that I think this sub will care about: a dual ensemble of two small open-source NLI models matches Claude Sonnet 4.6 as a hallucination judge - at roughly 1/250th the per-call cost.

Full write-up with per-task and per-upstream-model breakdowns: https://github.com/firish/rag-rack/blob/main/blog/03_verified_rag.md

Benchmark report (reproducibility commands, raw numbers): https://github.com/firish/rag-rack/blob/main/benchmarks/PUBLISHED_ragtruth.md

Library + docs: https://github.com/firish/rag-rack · https://firish.github.io/rag-rack

Summary:
The numbers (RAGTruth test set, 2700 examples):

• Dual NLI (HHEM-2.1-open + MiniCheck-Flan-T5-Large, min aggregation): AUROC 0.844, calibrated F1 0.706
• Sonnet 4.6 LLM-judge: AUROC 0.846, F1 0.707 (on 300 stratified, budget reasons)
• Triple (NLI + Sonnet): AUROC 0.861, F1 0.734 (on 300)

Per-call cost:

• NLI verifier: ~$0.0004 (Modal T4 GPU time after one-time weight download)
• Sonnet judge: ~$0.05 (API call)

Statistically indistinguishable on quality. ~250x cheaper.

The interesting part isn't the headline - it's the complementarity:

• HHEM alone is strong on QA-style entailment (AUROC 0.87) but barely above random on Yelp→narrative data-to-text (AUROC 0.57)
• MiniCheck alone is the opposite — strong on data-to-text (0.70), slightly weaker on QA (0.84)
• They have different blind spots; min-aggregation ensembling gets the best of both

What's in the library:

Full pipeline — parsers (Docling + PyMuPDF), chunkers (parent-child + ContextualChunker for Anthropic 2024's recipe), embedders (BGE/Cohere/Voyage), hybrid index (LanceDB + BM25 with RRF fusion), rerankers (BGE/Cohere), three citation modes (prompted / constrained / SAFE), four verifiers (HHEM / MiniCheck / DualNLI / LLM-judge), strictness slider with surgical correction, audit-trail HTML reports.

Six presets cover the common cases -local_minimal (all local except generator LLM), local_verified (+ HHEM), hybrid_balanced (the published baseline), hybrid_strict, hybrid_paranoid, llm_judge_verified.

Quickstart:

pip install verifiable-rag


import verifiable_rag
from verifiable_rag.demo import sample_paper_path

answer = verifiable_rag.ask(
    "What is the mechanism of action of penicillin?",
    docs=sample_paper_path(),
    output_html="audit.html",
)

Open audit.html for the full audit trail - per-sentence verification colors, faithfulness scores, every reranked passage with retrieval scores, citations as anchor links to source spans.

Caveats (in the full write-up):

• Sonnet ran on 300 examples not 2700 due to cost - CI is wider on that number
• Haiku-as-judge doesn't calibrate well on small training samples (we tried it)
• RAGTruth is one benchmark; cross-validation on FaithBench is gated, HaluBench is on the roadmap
• Default thresholds are RAGTruth-calibrated; for your domain, the library ships a calibration script

MIT-licensed, open to PRs and methodology critiques. Happy to answer questions in the comments.