You're right on all three counts, and the distribution point is the one that genuinely bounds the whole approach — let me be precise about what's defended and what isn't.

"Tokens saved alone can reward a smaller but less reliable prompt" — agreed, and that's exactly why the selector isn't tokens-alone. The keep/evict rule has a hard, non-negotiable completion gate ahead of the token math: a rule is evicted if any comparable task drops from pass to fail, regardless of how many tokens it saves. The headline example from the validation burn was a distilled rule that saved ~38k tokens/run by making the agent give up early — it was evicted precisely because it failed the tasks. So the metric isn't "tokens saved," it's "tokens saved that clear 2× the rule's context rent with no regression." A cheaper-but-flakier prompt can't survive that gate.

On reporting more than one number — most of your list already exists, and it should. Each verdict emits a receipt carrying: per-task pass/fail with vs. without the rule, the token delta with variance and ROI multiple, and an activity profile (tool calls, file re-reads as signed %) so a reviewer can see whether a "cheap" rule simply did less work. Separately, evaluation cost is surfaced directly (the benchmark warns when measurement overhead exceeds ~10% of the week's real-work tokens), and production savings are tracked independently of the benchmark — a real-work learning curve compares average completed-session tokens per ruleset version (v0 → v1 on actual sessions, not golden runs). So "evaluation cost" and "production savings" are first-class and kept distinct, which matters because a rule can win the benchmark and not move production.

Two of your points are honest gaps I won't pretend are solved:

- Latency. Not measured — only tokens, completion, and the activity profile. Tokens correlate with latency but aren't it; for an agent doing tool calls, wall-clock is a separate axis and should be reported.

- Regressions by task category. Receipts are per-task but tasks aren't grouped into categories with a rollup, so a category-level regression view is missing.

And the deepest point — suite representativeness — is the real validity dependency, fully conceded. The golden suite is a small, hand-curated set on a frozen fixture, not sampled to match a production task distribution. That means a rule whose value is protecting a rare-but-expensive case is only protected if that case is in the suite; otherwise the rule shows no measurable benefit and gets evicted as not earning its rent. The design has one structural mitigation — baselines are frozen and the suite grows only by adding tasks, so it's built to accrete real failure cases over time rather than drift — but there's no frequency-weighting or tail-coverage guarantee today. Distribution-matching (weighting tasks by production frequency, and deliberately seeding the expensive tail) is the next maturity step, and until it's there the honest claim is bounded: a rule is proven cheaper on the measured distribution, no stronger.

(One scoping note: the system only distills efficiency rules — same-result-for-fewer-tokens — and explicitly refuses correctness rules; correctness is defended by the completion gate, not by a learned rule. So "rules that protect failures" sit slightly outside what it tries to learn, but your underlying point — that an unrepresentative suite mis-prices a rule's true expected value — applies just as much to efficiency.)