The non-inferiority swap: how we ship model changes on a quality tie

On 2026-09-01 we moved our paid think chat surface from Grok 4.6 (reasoning on) to GLM 5.3 (effort high) after an eval in which the challenger scored 1.7 points lower on answer quality. On our production prompt, the incumbent's first content token took a median 88,906 ms; the challenger's took 4,169 ms. The same eval also moved the paid fast surface (an outright +10.0-point win) and both modes of the free surface (+13.3 and +11.7 points). All three surfaces run the challenger in production today, with the previous models one env variable away. We call the practice behind these calls the non-inferiority swap, and we think it is the correct shape for production model changes: the eval's job is not to find a winner, it is to establish, under rules frozen before the run, that the challenger does not lose, and then let the dimensions users actually feel (first-token latency, tails, reliability) break the tie.

Why the intuitive gate fails

Most teams gate a model swap the obvious way: run an eval, ship only if the new model scores higher. Two things break that gate.

First, on real product tasks, quality differences between current frontier models are usually smaller than your judge's ability to measure them. Our 20-task compliance benchmark measured its judge's test-retest wobble at 18.2 points on a 100-point scale, and pre-registered its verdict as a tie on exactly that basis (the run happened the day after this swap; we wrote it up here). A gate that demands improvement on that instrument almost never fires honestly. A gate without one ships on noise.

Second, a strict-improvement gate creates the wrong incentives: either nothing ever ships, or somebody re-rolls the eval until the noise lands in the challenger's favor.

The alternative is borrowed from clinical trials. A non-inferiority trial does not ask whether the new treatment is better, only whether it is acceptably not worse, on a margin defined before any data exists. Borrow the design wholesale: pre-register a tie band and the operational guards, run once, and make the verdict mechanical.

The setup: three surfaces, three frozen rules

We evaluated three production chat surfaces of the same product, one eval harness, one run. Each surface got its own decision rule, frozen on 2026-09-01 before any call was made:

Surface	Incumbent	Challenger	Pre-registered rule
Paid fast	GLM 5.2	GLM 5.3-Flash, effort low	Quality within 5.0 points AND first-token p50 not materially worse AND p90 under the 8 s fallback race AND >10 s tail rate not worse by more than 5 pp
Paid think	Grok 4.6, reasoning on	GLM 5.3, effort high	Quality within the 5.0-point tie band earns the flip
Free (logged-out and over-quota fallback)	GLM 4.7	GLM 5.3-Flash, effort low	Quality within 5.0 points AND first-token p50 under 2x incumbent

The tie band was 5.0 points on the quality metric, frozen in the plan, not adjusted after seeing scores. One guard was frozen in prose rather than as a number: the paid-fast rule says the challenger's first-token p50 must be "not materially worse", with no threshold attached. The plan is the plan, and that guard required a judgment call in the results section; the fix is a numbered guard next time, and it is in the checklist. Our streaming layer falls back to a waiting state after 8 seconds without first content on fast surfaces and 120 seconds on think surfaces, so the latency guards were written against those real budgets.

Method, as frozen:

• The prompt was the production assembly. 22,445 characters: the concise style prompt, the framework-knowledge injection built from the question, the workspace block, the date. Byte-identical across all arms. A compact demo prompt would have measured a different product.
• Four tasks, three long-running GRC answer tasks plus one chat-native workspace-grounded policy task, each scored on a five-criterion 0-2 rubric. Three samples per arm per task at temperature 0.
• One judge model scored every output blind: Grok 4.3, run through OpenRouter, no arm labels, no knowledge of which model produced what. It shares a vendor family with exactly one arm, the think incumbent. Self-preference in LLM evaluators is documented when the judge scores its own generations (Panickssery, Bowman and Feng, arXiv:2404.13076, 2024-04-15), and position and verbosity biases are catalogued in LLM-as-a-judge setups (Zheng et al., arXiv:2306.05685, 2023-06-09). Neither paper measures family-level bias, and neither did we, so we recorded the direction of the risk instead of assuming neutrality: if family favoritism exists here, it inflates the incumbent, which biases the think flip conservative, not permissive.
• Incumbent baselines were regenerated fresh in the same run. Our June eval of the same incumbent used a different judge and an earlier prompt assembly, so reusing its stored numbers would have compared two different instruments. Same judge, same day, same assembly, or the comparison is not decisive.
• A pre-registered spend ceiling with an automatic abort, and hard per-call timeouts.

The results

Quality is the judge's mean score as a percentage of rubric maximum; 12 scored generations per arm (4 tasks, 3 samples), 2026-09-01:

Surface	Arm	Quality
Paid fast	GLM 5.2 (incumbent)	81.7
Paid fast	GLM 5.3-Flash, effort low	91.7
Paid think	Grok 4.6, reasoning on (incumbent)	91.7
Paid think	GLM 5.3, effort high	90.0
Free fast	GLM 4.7 (incumbent)	80.0
Free fast	GLM 5.3-Flash, effort low	93.3
Free think	GLM 4.7, reasoning on (incumbent)	80.8
Free think	GLM 5.3-Flash, effort low	92.5

Latency is time to first content token on a streaming connection; six reps per arm on one workspace-grounded production task (the same 22,445-character prompt assembly as the quality bench), 2026-09-01:

Arm	TTFC p50	TTFC p90	Reps over 10 s
Paid fast: GLM 5.2	773 ms	2,626 ms	0 of 6
Paid fast: GLM 5.3-Flash	2,641 ms	4,011 ms	0 of 6
Paid think: Grok 4.6	88,906 ms	101,591 ms	6 of 6
Paid think: GLM 5.3	4,169 ms	57,707 ms	2 of 6
Free fast: GLM 4.7	2,857 ms	3,163 ms	0 of 6
Free fast: GLM 5.3-Flash	3,350 ms	3,640 ms	0 of 6
Free think: GLM 4.7	18,014 ms	24,156 ms	6 of 6
Free think: GLM 5.3-Flash	3,052 ms	7,612 ms	0 of 6

Applied mechanically, the frozen rules gave three verdicts:

• Paid fast: GO, and the surface got slower. Quality +10.0 points. First token 3.4x slower (773 ms to 2,641 ms median), but p90 at 4.0 s against the 8 s race and zero reps over 10 s on both sides. This is where the verdict stopped being fully mechanical: the prose guard ("not materially worse") has no threshold, and we judged that a 2.6 s median with a 4.0 s p90 and no tails clears it. That call is recorded here rather than laundered into a threshold that never existed. Whether users perceive 2.6 s as worse than 0.8 s is not something this eval measured; what it established is that the guard as frozen passed, and that a faster-than-incumbent requirement would have vetoed a swap the frozen rules scored as an outright quality win.
• Paid think: GO on a tie. 90.0 against 91.7 is -1.7 points, inside the 5.0-point band. Against that, the incumbent's 88.9-second median first token on our probe (6 of 6 reps over 10 s) became a 4.2-second median (2 of 6 over 10 s, and the incumbent's p90 was already past 100 s). A 21x improvement in the number users feel, for 1.7 points that sit inside the pre-registered tie band.
• Free: GO. +13.3 points in fast mode, +11.7 in think mode, and the old think arm's 18.0-second median with 6-of-6 tails became 3.1 s with none.

The think verdict is the one worth arguing about, so here is the argument in full. The pre-registered band says a 1.7-point gap is not a quality difference. The 18.2-point retest wobble our compliance benchmark measured the next day, on a different task set and rubric, is the only direct measurement we have of judge resolution on our instruments, and it sits an order of magnitude above this gap. Neither number is a confidence interval on this particular gap, and the band is a decision rule, not a measured noise floor. But both point the same way. Holding the think surface to 1.7 points that our instruments do not resolve, while the incumbent's median first token on our probe sat at 89 seconds, inverts the priorities: the number users feel first is latency, and it moved by 21x in our measurement.

A verdict has a shelf life

In June 2026 we evaluated GLM 5.2 (released 2026-06-16) as a replacement for GLM 4.7 across the same product, and the verdict on 2026-06-18 was do not replace: two independent blind judges scored it a quality peer of GLM 4.7, with exact ties once an infrastructure error was excluded, but it failed the operational gates decisively, most of all on think latency: a median 93 s to first content token against GLM 4.7's 2.1 s on our tasks and configuration, a 44x gap.

Three months later its successor beat the same incumbent outright in both free modes of the September eval. Two conclusions we now treat as standing rules.

Model verdicts have a shelf life measured in months. A "quality peer, operationally unshippable" call is a verdict about one model at one point in time, not about the slot; the slot got a different model and the verdict changed. This is the other reason we regenerate incumbent baselines in every swap eval instead of quoting stored ones: stored baselines silently compare across instruments, and stale verdicts silently compare across model generations. The June number was true. We did not inherit it, and it did not bind.

Two things the harness caught

Empty generations at a tight cap. The first latency pass on 2026-09-01 ran every arm at a 256-token completion cap. All twelve rows of the two GLM think arms (GLM 4.7 and GLM 5.2, six each) came back empty: the reasoning channel consumed the budget before the first content token existed, so the row measured the cap, not the model. The Grok 4.6 think arm streamed fine at the same cap. We invalidated the pass and re-ran the GLM think arms at a 4,096-token cap before reading any percentile; the quality bench had used a 16k budget throughout and was unaffected. This is the same failure class as the silent zero we wrote about in August (data with nothing in it, scored as a measurement). The cheap guard is unglamorous: count empty generations per arm, and refuse to aggregate any arm whose empty count is nonzero.

The incumbent's request shape does not carry over. We probed whether the previous configuration could be copied onto the challenger. In our 2026-09-01 API-side testing, 24 of 24 calls across GLM 5.3 and GLM 5.3-Flash that carried reasoning: {enabled: false} on the route we tested returned HTTP 400, so in our probes, reasoning cannot be switched off on this family; effort low is the floor, not a knob. A model migration re-derives the request configuration from the challenger's actual behavior. Every parameter you copy from the incumbent without re-testing is a confound you put in your own eval.

What this does and does not claim

The quality numbers come from 12 scored generations per arm (4 tasks, 3 samples, temperature 0), one judge model, no human calibration of the chat rubrics, one run on one day (2026-09-01), on our prompt assembly and serving configuration only. The 5.0-point tie band was a pre-registered decision rule, not a measured noise floor for this instrument; our 20-task compliance benchmark, run the next day (2026-09-02) on a different task set and rubric scale, measured 18.2 points of judge test-retest wobble, so we treat quality differences inside the band as unmeasured, not as zero. Latency percentiles come from 6 streaming reps per arm on one workspace task, our prompt assembly, our configuration, that day: properties of our setup, not vendor benchmarks. All results are point-in-time as of 2026-09-01. A public writeup of our benchmark methodology, including how we measure judge wobble, is on the model quality page.

The checklist

Before your next production model swap:

1. Freeze the decision rule per surface before the run: tie band, latency guards, tail guards, spend ceiling, and what a tie means.
2. Size the tie band from your judge's test-retest wobble on your rubrics. If you have never re-judged identical answers, you do not know how small your measurable quality differences are.
3. Evaluate on the production prompt assembly, byte-identical across arms. A demo prompt measures a product you do not run.
4. Regenerate incumbent baselines in the same run, same judge, same day. Stored baselines are comparisons across instruments.
5. Guard tails, not means: p90 first-token latency and the over-10 s rate, against your real streaming fallbacks.
6. Count empty generations per arm and refuse to score any arm that has them. Fix the completion budget, then measure.
7. Re-derive the request configuration for the challenger, reasoning parameters included. Never copy the incumbent's shape into the eval.
8. Know which direction your judge's family bias pushes the decision, and design so it pushes conservative.
9. Put an expiry date on every verdict. Re-run anything older than a quarter.

A tie is not indecision. It is the verdict your quality eval can actually defend, and making it pre-registered is what hands the decision to the numbers your users feel.