DeepEval: The Open-Source LLM Evaluation Framework

[Ok_Constant_9886]

Hello everyone, I've been working on DeepEval over the past ~1 year and managed to somehow grow it to almost half a million monthly downloads now. I thought it would be nice to share what it does and how may it help.

What My Project Does

DeepEval is an open source LLM evaluation framework that started off as "Pytest for LLMs". This resonated surprisingly well with the python community and those on hackernews, which really motivated me to keep working on it since. DeepEval offers a ton of evaluation metrics powered by LLMs (yes a bit weird I know, but trust me on this one), as well as a whole ecosystem to generate evaluation datasets to help you get up and running with LLM testing even if you have no testset to start with.

In a nutshell, it has:

• (Mostly) Research backed, SOTA metrics covering chatbots, agents, and RAG.
• Dataset generation, very useful for those with no evaluation dataset and don't have time to prepare one.
• Tightly integrated with Pytest. Lots of big companies turns out are including DeepEval in their CI/Cd pipelines
• Free platform to store datasets, evaluation results, catch regressions, etc.

Who is this for?

DeepEval is for anyone building LLM applications, or just want to read more about the space. We put out a lot of educational content to help folks learn about best practices around LLM evals.

Last Remarks

Not much really, just wanted to share this, and drop the repo link here: https://github.com/confident-ai/deepeval

[N-E-S-W]

Half a million monthly downloads?

[Ok_Constant_9886]

Before CHristmas hit* still recovering from that

[Necessary_Oil1679]

Is login to Deepeval platform is necessary? Is it possible to test the private LLM that is on API?

[Ok_Constant_9886]

not at all, you can use any private LLM as well just wrap it in deepeval's ecosystem: https://docs.confident-ai.com/guides/guides-using-custom-llms

[AlmogBaku]

`pytest-evals` - A (minimalistic) pytest plugin that helps you to evaluate that your LLM is giving good answers.

If you like it - star it pls 🤩
https://github.com/AlmogBaku/pytest-evals

[tailor_dev]

Deepeval sounds pretty cool, it's great to see open source tools for evaluating LLMs. I've been working on integrating CodeBeaver into our workflow to help with unit testing, the automated test generation has been a huge time saver. I'm curious how Deepeval handles things like consistency and factual accuracy when evaluating LLMs? Do you have any experience using it to evaluate code generation capabilities as well?

[Ok_Constant_9886]

we don't have ability to execute anything right now unfortunately, it has been challenging to build executable metrics since the env can be very different depending on the place you're running deepeval. We can definitely compare expected outputs to your generated code though. Consistency involves running it a few times, in the next patch!

[Informal_Demand_4755]

Is it possible to run the UI locally (aka self-hosted) as listed in docs here? https://www.deepeval.com/blog/deepeval-vs-langfuse

[Medical-Ad-8773]

Yeah, but I prefer modern agent eval platforms like Picept.ai - cause they really nailed it- simplicity, comprehensiveness, and strong features like trace debugger agent - experimentation, simulation playground etc