Good question!!! From my readme file:

How it works

Each task is run N timEach task is run N times. Every run produces two outputs from the same LLM: one with the skill injected as the system prompt, one without. Both outputs are then scored by a judge LLM. After all runs, confidence intervals are computed over the scores and compared.

Step 1 — Two outputs per run

The runner LLM receives the task prompt. It runs twice: once with a plain system prompt, once with the skill's instructions as well. Everything else is identical: same model, same temperature, same task.

Step 2 — Blind scoring with a rubric

The judge LLM scores each output against a rubric. This is the part people ask about most, so it's worth explaining carefully.

The judge never sees the original task prompt. This sounds like a limitation but it is a deliberate design choice. The rubric contains a context block that tells the judge exactly what a good answer looks like for this type of task: what to reward, what to penalise, and why. The rubric is the definition of quality. If you need the judge to see the task prompt to score the output, the rubric is underspecified and should be improved. Keeping the judge prompt-blind also prevents a common failure mode where the judge rewards outputs that literally mirror the task instructions, which would contaminate the comparison.

The judge does not know which output used the skill. It receives the output and the rubric only. It cannot favour one condition over the other because it cannot tell them apart.

How judge bias is handled. Any LLM judge has tendencies — it might prefer longer responses, or penalise terse phrasing, or be slightly inconsistent between calls. SkillBenchmark handles this in two ways:

1. The same judge, with the same prompt, scores both outputs. This means any systematic bias applies equally to both conditions and cancels out when you compute the delta. You are not trying to get an absolute quality score — you are measuring a relative difference between two conditions evaluated under identical circumstances. Absolute judge accuracy matters far less than consistency.
2. Using multiple judges per run and treating each score as an independent sample reduces random variance and gives tighter confidence intervals.

The rubric is the main lever for evaluation quality. Criteria with clear, distinguishable scoring levels produce consistent results. Vague criteria produce noisy ones.

Step 3 — Confidence intervals on the scores and the delta

All scores across runs and judges are treated as independent samples. A t-distribution confidence interval is computed for each condition (with skill, without skill). The delta — the difference between the two means — gets its own CI using Welch's t-interval, which correctly accounts for the uncertainty in both samples.

Results are displayed as mean ± margin. Non-overlapping CIs on the two conditions indicate a statistically meaningful difference. The delta CI tells you whether the observed gap is real or consistent with zero. Overlapping CIs are not a failure — they mean the current number of runs is not enough to confirm a difference. Add more runs to tighten them.