The silent zero: when a missing judge score becomes a measurement

The most dangerous number in an LLM-judged eval is not a biased score. It is a missing one, because a permissive aggregator turns it into a zero without telling you, and you then build a theory to explain the zero. We know because we did it twice in the same eval. In an offline ablation concluded on 2026-07-17, we compared ten generation strategies for GRC deliverables (policy packs, a processor RoPA, a SOC 2 gap analysis, an ISO/IEC 42001 AI policy, an EU AI Act classification), scored blind by two judge models on a 1-10 rubric. Pass 1 reported that rendering a whole deliverable in one pass beat rendering it step by step by 3.25 points. Pass 2 watched that effect collapse to 0.66, and our own methodology note diagnosed the collapse as a judge artifact: one judge model was "1.5-2 points more lenient" than the other. Both findings were the same bug. Seventy-nine of the 446 judge rows in pass 2 had no overall field, the reducer did Number(r[d]) || 0, and every overall mean in the report's scorecard, plus both framework-heavy headline figures, is reproduced exactly by averaging those 79 rows as zeros. We call this the silent zero, and this post is what it cost us and what we check now.

The eval, and what it reported

The setup was ordinary for this kind of work. Nine frozen scenarios, all synthetic and built around a fictional 20-person Berlin SaaS company (a tenth, a vendor due-diligence memo, sat in the fixture file but was not rendered in the passes analysed here), drawn from the hardest shapes of compliance writing we see: a multi-section internal audit report, a multi-document policy pack, a GDPR Art. 30 processor register, framework-heavy tasks on SOC 2, ISO/IEC 42001 and the EU AI Act, a workspace-grounded policy, a twelve-document run, an ambiguous-scope brief. Ten arms varied the generation strategy: render a pre-approved section-by-section brief step by step, render the same brief in a single pass, regenerate the brief with a stricter planner first, write freely from the client task with no brief, add a draft-critique-revise reflection loop, and swap the writer model. The two writer models were z-ai/glm-5.2 and anthropic/claude-opus-4.8, both called through OpenRouter in July 2026. Up to three trials per scenario and arm in pass 2; two arms were only partly populated (the regenerated-brief arm at 18 renders and the offline per-step rerun at 17, against 27 for the other newly rendered arms), and the captured baseline is a one-off reference set of nine documents.

Judging was blind and absolute: each judge saw the client task and one document, never the arm or the writer model, and was instructed to return strict JSON with five sub-scores (correctness, completeness, depth, citations, honesty), an overall, a list of fabrication flags, and a one-line verdict. Two judge models, the same two as the writers. Pass 1 used an anti-self-preference assignment, the GLM judge for Claude-written arms and the Claude judge for GLM-written arms, because self-preference in LLM evaluators is documented: Panickssery, Bowman and Feng showed that LLM evaluators recognize and favor their own generations (arXiv:2404.13076, 2024-04-15), and Zheng et al. catalogued self-enhancement, position and verbosity biases in LLM-as-a-judge setups (arXiv:2306.05685, 2023-06-09). Pass 2 applied both judges to every arm and averaged them, which is a better design.

Pass 1's headline, from the scorecard: the focused single-pass render scored 7.56 overall against 4.31 for the per-step render of the same brief on the same model. A 3.25-point topology effect. Pass 2's headline: that effect was 0.66, the free write from the task was the best arm, and the per-step design was near the bottom on quality and the worst on honesty and fabrication. The report's methodology note explained the gap between the passes: judging different arms with different judges had made absolute scores incomparable, because the GLM judge was roughly 1.5-2 points more lenient than the Claude judge. "Never compare absolute LLM-judge scores across arms judged by different models" was pinned as the lesson. It sounded right. It was not what the data said.

What the rows actually contained

On 2026-08-19 we re-read the raw judgments file, row by row, instead of the scorecard. Of 446 unique (scenario, arm, trial, judge) rows recorded in pass 2 (out of 466 planned; 20 planned judgments have no row at all, which the harness logged as failures and correctly left out of every mean), 79 had no overall key, 18%. They split two ways. Sixty-five rows were complete judgments minus one field: all five sub-scores present, fabrication flags present, the one-line verdict present, and no overall. The judge had returned the JSON we asked for with one key dropped, the key that happened to sit between honesty and fabrication_flags in the requested schema. Fourteen rows were JSON objects with no rubric fields at all, which the parser accepted because it accepted any JSON object. The missingness was not evenly distributed across judges: in our run, 56 of the Claude judge's 229 rows (24%) had no overall, all of them the partial kind, and 23 of the GLM judge's 217 rows (11%) had none, 9 partial and the 14 empty objects. We did not investigate why; our judge prompt, the JSON-mode handling on the route we used, and the 1,200-token output cap are all candidates, and the rate is a property of our call, not a claim about either vendor.

The aggregator then did the natural TypeScript thing:

for (const d of dims) out[d] = Number(mean(js.map((r) => Number(r[d]) || 0)).toFixed(2));

A row without overall became a 0 in a column where every real value sat between 1 and 10, and the valid distribution was clustered hard at the top (of 367 valid overall scores, 299 were 8 or higher). One zero in a cell of sixteen moves the mean by about half a point at the score levels we observed. Eight zeros in a cell of sixteen halve it.

Here is the pass-2 scorecard as published, next to the same rows with missing scores treated as missing. Both judges, all scenarios, overall on 1-10:

arm	published overall	valid-row overall	valid judge n / judge rows
per-step brief render, captured baseline outputs	6.56	7.50	14 / 16
per-step brief render, offline rerun (GLM-5.2)	6.12	8.08	25 / 33
single-pass brief render (GLM-5.2)	6.78	8.07	42 / 50
single-pass brief render + reflection (GLM-5.2)	7.02	8.14	44 / 51
regenerated brief + single-pass render (GLM-5.2)	6.32	7.68	28 / 34
free write from task (GLM-5.2)	6.38	8.30	40 / 52
free write + reflection (GLM-5.2)	6.08	7.71	41 / 52
single-pass brief render (Claude Opus 4.8)	7.45	8.59	46 / 53
single-pass brief render + reflection (Claude Opus 4.8)	6.56	8.53	40 / 52
free write from task (Claude Opus 4.8)	7.53	8.49	47 / 53

Every published overall in this table is the zero-filled mean to two decimals. The ten arms that appeared to span 6.08 to 7.53 actually span 7.50 to 8.59, a band of about one point, and the top four arms sit within 0.3 of each other. The published framework-heavy result, the one the report leaned on hardest ("Claude free 8.06 versus brief 6.59; GLM free 7.19 versus brief 6.40"), becomes 9.06 versus 8.62 (16 and 13 valid rows) and 8.85 versus 8.73 (13 and 11). A 1.47-point effect is 0.44; a 0.79-point effect is 0.12.

Pass 1 is the starker case. The per-step arm's 4.31 came from sixteen Claude-judge rows of which eight had no overall; the eight that did averaged 8.62. The focused arm's 7.56 came from eighteen rows with two missing; the sixteen valid ones averaged 8.50. The 3.25-point topology effect that launched the whole investigation was, on the rows that carried a score, minus 0.12.

The theory we built on top of the bug

This is the part worth internalizing, because the bug itself is dull. When pass 2 failed to reproduce pass 1, we did what careful people do: we looked for a mechanism. The anti-self-preference design meant the Claude-written and GLM-written arms in pass 1 had been scored by different judges, the documented biases of LLM judges made "the judges differ in severity" plausible, and a scorecard where the Claude-judged arms sat low and the GLM-judged arms sat high made it look measured. So we wrote down a leniency offset of 1.5-2 points and a rule about never comparing absolute scores across judges.

The raw rows let us test that theory directly, because pass 2 had both judges score the same document. Across the 148 (scenario, arm, trial) pairs where both judges returned an overall, the mean GLM-minus-Claude difference was +0.12. Seventy-three pairs were identical, 41 had GLM one point higher, 17 had Claude one point higher, and 17 differed by two or more. In the pass-1 data the same paired offset was minus 0.06 over 72 pairs. Among the 148 complete pairs there is no sign of a 1.5-point leniency gap. What there was: missingness, in two layers. Across judges, 24% of one judge's rows had no overall against 11% of the other's, and the anti-self-preference assignment put the judge with the higher rate on every GLM-written arm, so as a family those arms collected more zeros, which pushed their scores down in cross-family comparisons. Within a judge, missingness varied by arm: the two arms in the pass-1 headline were both GLM-written and both Claude-judged, and they had eight versus two rows without overall. The 3.25-point "topology effect" was never a judge comparison at all. We had read missingness as severity, and the explanation was satisfying enough that nobody opened the file.

That is the general shape of the silent zero. A missing value coerced to a number does not look like an error; it looks like a low score, and low scores get explanations. A judge with a constant severity offset is a much gentler failure: the offset cancels in any within-judge comparison and shows up immediately in a paired check. A zero-filled cell is not consistent. It scales with however often the parse failed in that cell, and in our data that rate was correlated with the arm.

What survived

Not everything in the report was wrong, and the split is instructive. The sixty-five partial rows still carried honesty and fabrication_flags, so those two columns lost only the fourteen empty rows. Re-derived on rows with a valid honesty value, the captured per-step baseline still has the lowest honesty (5.81 across 16 judgments) and fabrication flags on 16 of 16, while the Claude free-write arm has the highest honesty (9.75 across 52) and flags on 1 of 52. That gap is real. The quality ranking mostly dissolved into a one-point band, and the claim that "free writing beats brief rendering" went from a headline to +0.23 overall for GLM-5.2 (8.30 against 8.07) and -0.10 for Claude Opus 4.8 (8.49 against 8.59). The "reflection hurts" claim held for the GLM free write (7.71 with reflection versus 8.30 without) and was a wash for the Claude brief render (8.53 versus 8.59). We have not re-run the ablation; the conclusions the report drew are being re-derived from the valid rows, and the methodology lessons we actually learned are below.

There is a compliance reading of this too, and it is not decorative. ISO/IEC 27001:2022 Clause 9.1 b) requires an organization to determine the methods for monitoring, measurement, analysis and evaluation in a way that ensures valid results, and says the methods selected should produce comparable and reproducible results to be considered valid (ISO/IEC 27001:2022, Clause 9.1, 2022-10 edition). An internal auditor who found that an ISMS performance dashboard had been averaging blank cells as zero could raise a nonconformity against that clause. An eval scorecard that drives an architecture decision is, by analogy, a measurement method in the same sense, and we held it to a lower standard than we would hold a customer's.

Limits

This is a re-aggregation of the judgments we already had, not a new run, and the valid-row numbers inherit every limitation of the original: nine scenarios, up to three trials, integer scores from two LLM judges clustered at 8-9, no significance testing, and a judge that is itself a model. Valid-row cells run from 14 to 47 judgments per arm over all scenarios and from 4 to 16 on the framework-heavy subset; the valid scores have a standard deviation of 1.36, so a naive row-level standard error of a full-arm mean (1.36 over the square root of n, ignoring that judges are paired and trials cluster by scenario) is roughly 0.2 to 0.4, and we read differences under about half a point between arms as ties. The overall drop rates are what we observed on our prompt through one route in July 2026, at a 1,200-token cap with JSON mode requested; we did not isolate the cause and do not know whether it is stable. The paired judge offset of 0.12 is specific to this rubric and these documents. None of this changes the central fact, which does not depend on any of those caveats: the published overall column was computed with 18% of its rows coerced to zero, and the two findings built on it do not survive removing them.

The silent zero, and a checklist

The idea to carry out: in an LLM-judged eval, a judge call that returns no score is not a data point, and any aggregator that can turn it into one will eventually do so in the cell where it matters most. Judge-bias mitigations are worth doing, and they are worthless until you have counted your nulls. If you run an eval where a model grades model output, run this against it:

• Never coerce a missing score. Number(x) || 0, x ?? 0, fillna(0) on a score column: all of these turn an absent judgment into a score below the rubric's minimum, worse than any judgment the rubric allows. Treat missing as missing (pandas skips NaN by default in a mean; our hand-rolled reducer did not) and make the scorecard fail loudly if any cell's valid count is below what you pre-registered.
• Print valid n next to every mean. A cell that reads "4.31" and a cell that reads "4.31 (8/16)" are different claims. Ours printed n as the row count, which hid that half of one arm had no score.
• Validate the judge's output against the schema and retry. A JSON object is not a judgment. Require every scored field, reject and re-call on a miss, and log the miss rate per judge and per arm. A miss rate that differs by judge or by arm is itself a finding, and in our case it was the finding.
• Before you theorize about judge bias, compute the paired offset. If two judges scored the same documents, the difference on those pairs is the scoring offset between them on that task. Ours was 0.12 on complete pairs. A 1.5-point story that you did not get from paired rows is a story.
• Treat an implausible effect size as a data error first. A 3.25-point swing between two renders of the same brief on the same model, with scores clustered at 8-9 and sixteen to eighteen rows per arm, was implausibly large relative to the observed distribution and should have sent us to the rows before we looked for a mechanism.
• If you swap judges per arm, you swap their failure modes too. Anti-self-preference assignment is a defensible design, but any judge-specific defect (parse failures, truncation, refusals) is then confounded with the treatment. Running every judge on every arm is safer, and it still does not save you from zeros.
• Re-derive the headline from the raw file before the report writes a theory. The scorecard is a view. The rows are the evidence. Our report cited the scorecard, explained its own artifact with a mechanism, and pinned that mechanism as a lesson. The mechanism was a missing key.

All figures above are our own measurements on our own synthetic GRC fixtures (a fictional company, no customer data), writers and judges z-ai/glm-5.2 and anthropic/claude-opus-4.8 as served through OpenRouter for both rendering and judging, ablation concluded 2026-07-17, re-aggregated from the recorded judgments on 2026-08-19.