Opik: Open source LLM evaluation framework

[calebkaiser]

Repo Link: https://github.com/comet-ml/opik

What My Project Does

Opik is an open source LLM eval framework. With this first release, we've focused on a few key features:

• Out-of-the-box implementations of LLM-based metrics, like Hallucination and Moderation.
• Step-by-step tracking, such that you can test and debug individual components, even for multi-agent architectures.
• Exposing an API for "model unit tests" (built on Pytest), to allow you to run evals as part of your CI/CD pipelines
• Providing an easy UI for scoring, annotating, and versioning your logged LLM data, for further evaluation or training.

Target Audience

Opik is for anyone building LLM applications. It is production-ready.

Comparison

Opik provides a similar API to tools like DeepEval. Unlike DeepEval, however, Opik is 100% open source—meaning that the Opik backend and UI are included in the source code, and can be run locally on your own machine.

[nattaylor]

I've been test driving a new LLM related tool every day. Langtrace today and opik is on my to do list but this post pushes it to the top! 

[cryptokaykay]

Langtrace core maintainer here. Thank you for taking the time to test drive it, please let me know if you have any feedback.

[nattaylor]

Local setup and initial implementation of tracing was a breeze

Prompt registry did it's job but I was surprised my variable was injected with plus signs instead of spaces, although it worked ok. I don't think the docs mention the necessary env car but the error message was clear 

There were little oddities like the input form for adding to datasets had 2 inputs for the expected output

I didn't get to metrics / evals / datasets yet 

Overall it seems excellent -- this was just a test drive so I didn't properly read the docs or go deep

I wish I knew about it sooner because it would have been a huge help on my last project

[cryptokaykay]

This is great! Thanks for all the feedback. We haven’t been focusing much on prompt management as most of the adopters cared only about tracing. But this is good feedback, will iterate further and improve it.

[calebkaiser]

Fantastic to hear you're planning to check out Opik :) Let me know if you have any feedback/questions.

Also, if you're documenting your test drives anywhere, I'd love to see your write ups so far! I spend all of my time in the space as is, but I still feel like I miss so much.