The symmetry test: shipping a prompt fix you cannot reproduce

You cannot always prove a prompt fix works. Some production failures only appear with the full context that produced them, and no offline fixture reproduces them. When that happens, the honest question is not "does the fix succeed" but "is my change distinguishable from the model's own run-to-run variance." If the answer is no, the change is safe enough to ship even though you never once watched it fix the bug. We call the check the symmetry test, and it is how a prompt change we could never make fail cleared our gate.

Everything below comes from one internal evaluation, dated 2026-07-09, run against GLM-5.2, the model we run for chat: a blind A/B regression suite over our whole chat surface. The metrics are ours, measured on our own fixtures, on that date.

The bug that would not reproduce

In one week we saw two document-generation defects on German policy output. In the first, the assistant delivered a policy whose top-level sections were numbered 2 through 6 with no section 1. In the second, a referenced attachment never reached the model, and instead of saying so, the assistant fabricated a garbled "revised version" that the reader took for a massive truncation, and never disclosed that it had not actually seen the document.

We wrote a prompt rule for each: a numbering invariant (top-level sections start at 1 and stay gapless) and an anti-fabrication rule (if the referenced document is not present, ask for it, do not invent one). Then we tried to reproduce the original failures against the old prompt so we could watch the new rules fix them.

The numbering defect would not reproduce. Across more than twenty generations on the old prompt, including the production message shape that triggered the report, GLM-5.2 started at 1 every time. The failure did not appear without the fuller context the live session carried: accumulated user memories, a long multi-turn thread, and reasoning mode on. Our reading is that one or more of those is needed to trigger it. The single-turn offline harness did not have them, and the bug was simply not there.

This is the trap. We had a fix for a bug we could not summon. We could not show the fix improving anything, because there was nothing broken to improve in the harness. Shipping on faith is not evidence. Blocking forever because you cannot stage the failure ships nothing. Both are wrong.

Reframe: bound the blast radius, do not chase the fix

The rule lives in the system prompt on every chat turn, so its blast radius is not "document numbering." It is every behavior the assistant has: framework Q&A, cross-framework mapping, legal disclaimers, refusals, multilingual output, tone. A prompt edit aimed at one failure mode can silently dent an unrelated one. That is the risk worth measuring, and unlike the original bug, it is measurable offline.

So we stopped trying to prove the fix worked and started trying to bound the risk the change carried. The design: a blind A/B regression eval over the whole chat surface, comparing the old prompt against the new one on identical inputs.

• 42 fixtures spanning compliance Q&A, advisory, framework mapping, service redirects, safety refusals, legal disclaimers, multilingual German and French, tone, document generation, and dynamic fixtures carrying the custom-instruction and pinned-file prompt sections real users have.
• 252 generations, forming 126 old-versus-new pairs (three samples per fixture), all on GLM-5.2, 2026-07-09.
• Blinding. Each pair is written as A/B with a seed-randomized label. A manifest holds the true old/new mapping and is never shown to the graders.
• Two grading layers. A deterministic layer checks must and must-not conditions on the gated fixtures (help-center redirects fire, legal disclaimers fire, prompt-extraction attempts get refused, umlauts survive, minimum lengths hold). A blind qualitative layer has graders read each pair with no knowledge of which arm is which or what changed, and rule equivalent, A-better, or B-better with a materiality tag.

The deterministic layer came back clean: zero failing responses on either arm, zero asymmetric failures. Nothing the rule was supposed to protect broke on either side.

The qualitative layer is where the interesting move is.

The symmetry test

The blind graders rated 118 of 126 pairs equivalent. Of the remaining eight, seven carried a material difference and one was a minor, non-material difference. On a naive reading, seven divergences in a safety eval is a fail.

It is the wrong reading, and the reason is direction. A material difference has a sign: either the new arm was worse or the old arm was worse. If your prompt change is causing regressions, the material differences skew one way, toward new-worse. If they are the model's own run-to-run variance and have nothing to do with your change, they land on both arms at roughly equal rates. So do not just count the material differences. Un-blind them and count the direction.

We did. The seven split three new-worse, four old-worse. Three against four is as close to even as seven can fall, the pattern you would expect from run-to-run variance rather than a directional regression. None of the seven had any plausible connection to a document-numbering rule:

Material difference	Worse arm	What we observed
ISO 27001 clause identity	new	A.8.2 read as a risk-assessment control (correct in this fixture's other two samples)
PCI DSS segmentation	new	segmentation overstated as strictly required
PCI DSS segmentation	old	a fabricated "Appendix A1 segmentation requirement"
German document (CJK leak)	old	a stray Chinese token in German closing text
German document (CJK leak)	old	Chinese characters in a German summary section
Concision instruction	new	broke an "extremely concise" custom instruction
Concision instruction	old	broke an "extremely concise" custom instruction

The most vivid, our single sticky example, are the CJK leaks: in two of our German fixtures, the output carried a stray Chinese token (in one case the characters for "information security") in an otherwise German deliverable. That token-leak behavior has shown up on both arms across our German runs. Here both instances landed on the old arm. If the new rule were degrading German output you would expect them on the new arm; they were not there.

The regulatory-precision cases point the same way. In ISO/IEC 27001:2022 (published 25 October 2022), Annex A control 8.2 is "Privileged access rights"; one new-arm sample read it as a risk-assessment control, and got it right in the other two samples of that fixture. Under PCI DSS v4.0 (PCI Security Standards Council, March 2022), network segmentation is not a blanket requirement, it is a scope-reduction technique the Council describes as a way to take systems out of assessment scope, not a mandated control; one new-arm sample overstated it as required, and one old-arm sample invented that "Appendix A1" is a segmentation requirement set. Appendix A1 is actually the additional requirements for multi-tenant service providers. These are the kind of factual wobble a probabilistic model produces on hard domain questions, and in this run they fell across both arms, not concentrated on ours.

Meanwhile the behaviors the change actually targeted held on both arms throughout: every missing-attachment pair asked for the document instead of fabricating one, and every numbering pair produced a gapless structure. Across the full eval, the 126 blind pairs plus the deterministic sweeps over the same surface, we found no failure attributable to the change.

What this does and does not prove

Be precise about the claim, because it is easy to oversell. The eval shows no directional regression at this sample size, not that the fix works. We never demonstrated the numbering rule fixing the numbering bug, because the bug needs production context we could not stage. The rule ships on two legs: the production forensics that make the failure legible, and a non-regression signal showing the change does not disturb anything else. That is a weaker claim than "we reproduced the bug and watched the fix kill it," and it is the strongest claim the situation allows.

The limits are real and worth stating once, plainly. A gross directional skew would show at three samples per fixture; a subtle one would not, since a 55/45 lean hides at this n, and three-samples-across-42-fixtures is clustered, not 126 independent trials. We did not run an old-versus-old control arm, so the three-four split is our best estimate of baseline noise, not a measured baseline, and without an equivalence margin set in advance this is a no-directional-regression signal, not a formal non-inferiority test. The fixtures are single-turn, so an interaction between the new rule and a long production thread is out of scope, which is exactly the context the original bug needed. A symmetric split is evidence the change did not push the outputs one way, not proof it is inert. What the test buys you is a concrete read on blast radius when the alternative was a hunch.

The portable checklist

When you have a prompt fix for a production bug you cannot reproduce offline, do not ship on faith and do not freeze. Bound the change instead:

• Separate the two questions. "Does the fix work" needs the failure you cannot stage. "Is the change safe" does not. Answer the one you can.
• Size the blast radius honestly. A rule in a shared system prompt touches every behavior, not just the target one. Build fixtures across the whole surface, including the custom-instruction and pinned-context variants real users carry.
• Run it blind and paired. Same inputs, old versus new, seed-randomized A/B labels, and a manifest the graders never see. Blinding is what stops you scoring the arm you hope wins.
• Count direction, not just difference. Un-blind the material differences and check the sign. Asymmetric toward new-worse is a regression. A roughly even split across both arms is the pattern that clears a change at this level of evidence.
• Add a repeat control if the margin is close. An old-versus-old arm measures your baseline variance directly, so you are comparing your change against a number instead of an assumption.
• Keep a deterministic floor. Pair the qualitative read with hard must and must-not checks on the behaviors that must never break, so a symmetric-noise verdict cannot paper over a real broken invariant.
• State the claim you actually earned. Say "no measurable regression," not "the fix works," when non-regression is all you proved.

The instinct when a bug will not reproduce is to keep trying to stage it until it breaks. Sometimes it never will, and the effort is spent on the wrong question. Whether the fix ships turns on whether your change is louder than the model's own variance. Measure that directly, and a prompt edit you could never watch succeed becomes one you can still be confident is safe.