Engineering

Our eval's verdict was decided by a number we froze before the run

The tie band: we measured how much our judge wobbles when re-judging identical answers, froze it into the pre-registration, and let it overrule a 16.1-point gap. The same pre-registration caught a pilot that would have blinded the real run.

by ISMS Copilot··7 min read
Our eval's verdict was decided by a number we froze before the run

A 16.1-point gap, a confidence interval that excludes zero, and the verdict was still a tie. That is not statistical cowardice. It is the only honest verdict our instrument could defend, because we had measured, before the run, that our judge moves by 18.2 points one retest in ten when re-scoring identical answers. We froze that number into the pre-registration and let it overrule the headline. If you run LLM-as-judge evals and you have never re-judged identical answers, you do not currently know how small your measurable quality differences are, and every verdict you publish is bigger than your instrument.

The wobble number, and where it came from

On 2026-09-02 we ran a pre-registered 20-task compliance benchmark across six scaffolding configurations of the same model (GLM 5.3-Flash), with a cross-family judge model scoring every answer against frozen rubrics. The full results and losses are in our benchmark writeup. This post is about one number inside that design: the tie band.

The band is the 90th percentile of judge test-retest deltas. The procedure: take a stratified sample of answer cells across the arms and framework families, re-run the judge on the identical prompt and identical answer twice, and measure how much the weighted score moves. On our instrument, 12 retest pairs produced a 90th-percentile wobble of 18.18 points on a 100-point scale. The judge is temperature 0, pinned to a single serving provider, schema-validated, and blind to arm labels. Identical inputs. Eighteen points of movement, one run in ten.

That number became a frozen decision rule, written into the plan before any generation spend: WIN if the difference exceeds the band, LOSS if it is worse than the negative band, TIE otherwise. The band file carries a SHA-256 digest of the full shared instrument (tasks, arms, fixtures, prompts, configurations), and the run aborts at startup if any row carries a foreign digest. We did not choose 18.18 because it made a convenient verdict. We computed it from retests, froze it with its digest, and the verdict followed from arithmetic.

What the band decided

The pre-registered primary endpoint was the full production assembly against the better of two independently authored DIY prompts. The product scored 92.1%, the best DIY prompt 76.0%. Mean difference: +16.1 points, framework-clustered bootstrap 95% confidence interval [3.4, 30.2] across ten clusters and 2,000 resamples. The interval excludes zero, comfortably. The difference does not exceed 18.18.

Pre-registered verdict: TIE. Reported as a tie, headline and all, because that is the arithmetic. Two things are true at once: on this task set the direction of the effect is positive and unlikely to be zero, and on this instrument we cannot call the size of the effect a difference. A verdict that reports only the first truth is marketing. A verdict that reports both is a measurement.

This is the part almost every judge-based eval we have read skips: the verdict is a property of the instrument, not of the systems. An 18-point wobble means single-sample differences of 15 points are indistinguishable from re-running the judge on unchanged answers. Any leaderboard, comparison post, or ship gate that ignores its own retest wobble is publishing noise with a decimal point.

The pilot gate the pre-registration built

The pre-registration also did quieter work before the real run. Our methodology requires a pilot gate on a small task subset before any scaled spend: check parse rate, truncations, and that no compared arm is too saturated to discriminate. The first pilot attempt (run pilot-1, 2026-09-02) tripped that gate. The knowledge-armed auxiliary arm hit 100% mean while the compared arms sat between 52.7% and 83.3%. Score differences between compared arms were still separable, but the ceiling sat close enough that margins would compress, and we had frozen a refinement rule for exactly this case: the any-arm saturation criterion was revised to apply to compared arms only, the instrument was edited, and the pilot re-ran from scratch (pilot-2, same day, passed: 100% parse rate, zero truncations, 12 retest pairs, detection fired on every knowledge-bearing task). Only then did the full run start.

Two lessons in that sequence. First, pre-register the refinement path too: we changed the gate, but the change was itself pre-planned and re-piloted before the freeze, so the full run's pre-registration was never edited after data existed. Second, the arm that scored 100% in the pilot (the knowledge-armed DIY configuration) scored 96.9% in the full run, the highest arm apart from the knowledge-plus-documents comparison arm. Without the gate, the primary endpoint would have run against instruments drifting toward saturation, which would likely have compressed the 16.1-point gap, and the tie verdict would have arrived for a worse reason.

The published loss

The honest-interest test for a decision rule is whether it can produce a result you would rather not publish. Ours did. The pre-registered arm design included the production assembly, a knowledge-armed DIY arm, and a knowledge-plus-documents DIY arm as a secondary comparison. The knowledge-plus-documents arm outscored the product, 98.7% to 92.1%, with five task cells differing. We published it in the benchmark writeup, in plain text, in the losses section.

The band is what made publishing it cheap. A verdict system that only ever produces wins is not a decision rule; it is a press release with a rubric. Once the tie band and the loss-publishing rule are frozen before the run, a losing arm is just an outcome, not an embarrassment. That symmetry is the whole trick: the same rule that forced us to call a 16.1-point gap a tie also committed us to publishing a result where a hand-assembled prompt beat our product.

One concrete defect the run surfaced, for the record: the product arm failed one of the four trap questions, describing a nonexistent ISO 27001 control (A.8.35) with a full invented requirement, while its own injected control table in the same prompt ended at A.8.34. The knowledge-armed DIY arms refused the same false premise. We published that too.

Limits

This is one instrument, one day, one judge model (cross-family, temperature 0, single provider pin), 12 retest pairs, 20 tasks, one generation per arm per task. The 18.18-point band is a property of our rubric scale and our judge; your band will differ, which is the point of measuring yours. A tie band is a decision rule, not a confidence interval on any particular gap; a 90th-percentile retest delta is a conservative proxy for judge noise, not a measured sampling distribution. Human calibration of a graded sample, and a larger task set, are the next steps that would narrow what this instrument can resolve. None of these limits change the transferable claim: an eval that has not measured its own retest wobble cannot defensibly report differences smaller than that wobble.

The checklist

Run this on any judge-scored eval you own:

  1. Re-judge 8 to 12 identical answer cells across your arms and task strata, before your next real run. Compute the 90th percentile of the score deltas. That number is your resolution floor.
  2. Write the verdict rule against the floor into the plan: WIN / TIE / LOSS thresholds, loss-publishing included, before any generation spend. (If your plan also carries a saturation pilot gate, as ours does, keep it separate from the band: the band decides the verdict, the gate decides whether the run is measurable at all.)
  3. Freeze the instrument with a content hash and abort the run if any row carries a foreign digest.
  4. Add a pilot gate that refuses to scale when a compared arm's mean sits near the score ceiling, and pre-register how the gate may be refined (re-pilot from scratch, never re-run the full experiment against edited gates). Same rule for every other pre-run guard: if it can be revised, the revision path belongs in the plan, applied before the freeze, not after data exists.
  5. When the verdict lands inside the band, report the tie as the headline and the direction as context, not the reverse.
  6. Treat any reported difference smaller than your measured wobble, in your own posts or anyone else's, as unmeasured.

The most useful number in your eval harness is not a benchmark score. It is how much your judge moves when nothing changes.

Where to go next

Related Posts

Ready to do compliance work faster?

Built for speed, accuracy, and audit-ready output.