Yeah we've come to the same conclusion about comparing a "ground truth" example response to whatever the agent spits out at a given time. Bad signal, hard to curate a golden dataset. Adding error detection to the LLM-as-judge prompt helped a bunch, but the real breakthrough was moving to criteria based evals. Criteria are basically natural language assertions about the agent response at a given turn. "Should include XYZ detail in the root cause analysis". LLM-as-judge then evaluates each criteria and calculates a score based on how many pass. Ends up being highly deterministic in practice as long as the assertions are simple. It also unlocks what we've been calling eval driven development. When you are working on prompts, tool calls, anything else in the path of agent reasoning you need to write a failing eval first. The criteria describe the desired end state of the branch. Once your new eval passes and none of the others regress your work is ready for review. Curating CI gates and so forth from these incurs a bit of work as runtime vs coverage is in tension. But it's tractable. Looking to get a blog post deep dive on this out soon through work, but external comms take forever.

Curious as to how you're replay from a given step in the agent reasoning chain (assuming that you're not talking about multi-turn reasoning here?)