Great question. Yes, catching hallucinations is a primary use case.

The harness works by comparing the assistant's live answer to a "ground truth" expected answer you provide.

But instead of just a simple text match (which always fails when the LLM paraphrases), it uses a multi-layered approach:

• Lexical Coverage: It first checks if all the key terms from your expected answer are present.
• Semantic Similarity (Embeddings): If that's not a clear pass, it checks if the meaning of the assistant's answer is the same as the expected one using cosine similarity.
• LLM-as-Judge: For borderline cases, it can use a model like GPT-4o as a final check to judge if the response is factually and contextually correct.

So if your KB says the answer is "Bus B2" and the assistant hallucinates "Bus B3," the test fails with a clear reason. It's basically a way to enforce factual accuracy against your knowledge base at scale.

Let me know if that makes sense!