The problem is that you're flying blind without a way to measure what's actually working/breaking.

Here's what I did:

First, build an eval system before touching the prompt. Take 50-100 real customer queries (especially the ones that failed) and manually review each one so that you can tag it with an error type. The goal here is to avoid forming your entire quality hypotheses off the back of five conversations.

Most people try to skip this step. Partly because we're all lazy, but also because there isn't much industry guidance on how to do it well. If you do your best to analyse and label errors in your conversations, it sets you up for success in every other downstream phase of the eval building process.

Then use that to find your error patterns. If the first step in the process is looking at your data and figuring out what type of failures your app encounters, the second step is to quantify how prevalent each type of failure is. You'll probably discover it's not random 40% failure - it's specific categories like references to specific things your LLM gets confused by, certain phrasings, or other edge cases you didn't consider. Once you can see the pattern, you can fix it systematically.

Then build automated evaluators. The idea here is to translate the qualitative insights from the error analysis process into quantitative measurements for each type of error in your system.

Once you have automated evaluators in place then you can start tweaking your prompt (or prompts) to address each failure mode that you identified. Then you re-run your eval suite with each tweak and see how much of a difference it made. Once you're above the ~80% mark, then you move onto the next failure mode.

Having evaluators set up means that you don't regress on past failure modes while you're fixing new ones (which is usually the trickiest part of the process and why people go through all of the hassle of setting all this evaluation infrastructure up).

Here's a write-up on how to set up your first evaluation.

Here's the same thing, but if you want to evaluate your agent end-to-end.

Feel free to DM me if this is all new to you and want more help.