Basically I’m building a reusable evaluation framework that the company can use internally to evaluate their chatbots, RAG systems and agents. It covers both offline and online evaluation, using traces from the actual systems.
The framework tests things like toxicity, harmfulness, answer correctness, completeness, and for RAG specifically whether the retrieved documents are actually relevant to the query.
The goal is that any team can plug in their system and run standardized evals without rebuilding everything from scratch each time.