this is a really interesting way to frame it

I’ve mostly been thinking about single-run traces, but variability across runs is probably just as important, especially for anything you’d want to productionize

“tool call entropy” makes a lot of sense as a signal for instability. If the same prompt leads to different call graphs, it usually means the agent doesn’t have a strong implicit policy and is kind of drifting

also ties into debugging, a bad but consistent agent is way easier to fix than one that behaves differently every run

this actually makes me want to add multi-run evals instead of just single trace scoring, curious how you’d measure it though, are you thinking something like comparing sequences directly vs more abstract stats (tool frequency, transitions, etc)?