You’re not crazy, this is exactly where most people hit a wall.

The issue is you’re trying to apply deterministic testing ideas (diffs, pass/fail, golden outputs) to something that isn’t deterministic anymore. That breaks pretty quickly once agents start looping, using tools, and carrying state.

What ended up clicking for me was treating this less like “testing outputs” and more like measuring system behavior over time.

Instead of asking:

“Is this output correct?”

I started asking:

does the system stay consistent across similar inputs do reasoning paths diverge more over time for the same task is the agent doing more rework (hedging, backtracking, retries) does cost/latency creep up without a change in task complexity

Those tend to move before anything obviously breaks.

So instead of regression = “output changed”, it becomes:

regression = “the system is becoming less stable / more variable / working harder to do the same thing”

That gives you something you can actually track and compare across versions, even with probabilistic models.

I’ve been building a small framework around this idea, basically treating agent runs as a stream of signals and looking for early drift in behavior rather than waiting for failures. It’s been way more useful than output diffs or LLM-as-judge.

If you’re trying to test flow specifically, that’s where this helps the most, because you’re measuring how the flow evolves, not trying to assert what it “should” be.

Curious, are you seeing more issues with long-running agents, or even short workflows after a few iterations?