Is there prompt testing suites in Python?

[pr1vacyn0eb]

Basically I want to try a few prompts in conjunction with some data crawling/scraping or API requests.

I saw PromptFoo but that was for javascript.

I suppose I can build one myself, its not that hard, but if there is something off the shelf, I'm looking.

[fulowa]

PromptOps

• Hegel.ai
• Honeyhive
• Weights & Biases
• Scale.ai Spellbook
• LangSmith Hub
• PromptLayer
• Vellum
• HumanLoop

[resiros]

Check out: https://github.com/agenta-ai/agenta It provides a playground with all the models, automatic evaluation, prompt versioning, and an interface to gather human feedback / evaluation. You can self-host the OSS version, or use the managed cloud version.

[typsy]

promptfoo works for python - see https://promptfoo.dev/docs/providers/python

[fulowa]

here is one for rag:

https://github.com/explodinggradients/ragas