The blind verifier: what our anti-fabrication gate is allowed to see

In July 2026 we tested the verifier that guards our multi-step compliance pipeline against fabrication by planting fake facts in its inputs. Most of the planted fabrications sailed through. That result looked like a security hole in the one component whose job is catching fabrications. It was not, and the reason it was not is a design rule we now treat as a standing principle.

The plant test had fed its fake facts into the reasoning brief, the one document the verifier is contracted to treat as decided truth, and then counted how many the verifier flagged. Measured that way, the verifier could not have passed a competence test on fabrication at all. Meanwhile the repair everyone reaches for, letting the verifier flag any draft fact absent from the brief, did something worse than miss fakes: against clean drafts it flagged 9 of 9 on one model family and 3 of 9 on another. And the narrow contract we kept caught, on a real production run, the failure class its scope actually covers: a citation the draft asserted and the brief's approved set did not contain.

We call the principle behind this the blind verifier: an anti-fabrication gate must be scoped to what it can actually see. The one widening we measured judged truth without access to sources and flagged most clean drafts while catching none of the fakes that motivated it. Honesty checking has to be mechanical, and mechanical means the answer lives in the inputs.

The test that lied to us

The pipeline is Beyond, our multi-step document generation mode. A planner model reads the task and writes a reasoning brief. An executor model renders each planned step against that brief. A verifier model judges each step's draft before it is released to the user. (A fourth stage recaps the finished run; it plays no part in this story.) The three judgment roles receive different inputs, and that difference is the whole point of this post.

Our verifier is deliberately blind. The planner and the executor both see the full context: conversation history, workspace memories, workspace files, uploaded document content, and injected framework knowledge. The verifier sees two documents only, the brief and the draft under review. Its contract treats the brief's facts and judgments as decided, and its job is to check that the draft stayed faithful to them.

The lab test violated that premise without noticing. We inserted fabricated facts by hand into a bare brief, ran the verifier, and counted how many planted fakes it flagged. It flagged few; most of the planted fakes passed. We called the verifier broken. What the experiment actually measured was whether the verifier would second-guess the document it is contracted to trust. It declined, as designed.

In the same window, a production run surfaced what looked like the smoking gun: an organizational fact in released output that no input we checked contained. Before shipping a fix, we retracted it ourselves: the "fabricated" fact was in an uploaded file the executor had legitimately read. The fact was grounded. Our spot check, like the verifier, simply had not looked at that file. An investigator who cannot see the sources cannot judge groundedness either. We had run the same broken experiment on ourselves that we had run on the model.

The measured trap

So we turned to the draft side, where the verifier's inputs actually differ. The intuitive repair: flag any factual claim in the draft that does not appear in the brief. We built exactly that rule and measured it against clean drafts, drafts containing no fabrications at all, generated by the model families the pipeline ran at the time. The measurement and the decision to reject the rule are recorded in the shipped contract's design comment and commit history; the work was done and recorded in the 2026-07-22 review session:

Rule under test	Clean drafts flagged	Drafts generated by
Flag draft facts absent from the brief	9 of 9	GLM 5.2
Flag draft facts absent from the brief	3 of 9	Grok 4.20

A gate that rejects most or all clean drafts is not a gate; it is an outage. On GLM 5.2, every tested clean draft would at minimum have triggered a re-render; what a second failed verify would then have produced (a caveat flag or a halt) we did not measure. The mechanism is structural. A fact in the draft that is absent from the brief is not evidence of fabrication, because the executor is allowed, by design, to ground facts in sources the verifier cannot see. To judge "is this claim real?", a source-blind verifier has only its prior beliefs, and compliance work is the domain where priors fail in both directions: the true facts are specific to the customer's workspace, and the false ones are plausible.

Note what the widened rule also could not do: catch any fake from the plant test, because those fakes lived in the brief, which both the narrow contract and the widened rule treat as truth. By construction the widening added no recall on that class; our clean-draft measurements showed what it added instead. That pair of results is what convinced us the mandate, not the model, was the problem.

The contract that survived

What the verifier is allowed to judge is written into its shipped prompt, and the boundary is sharp. Per step, it enforces two contracts and nothing else:

1. The step's own contract: the draft delivers what this step's title and content spec require. Missing content is a violation only when this step's own spec calls for it.
2. The brief's judgment and citations as global constraints, where exactly three things are violations:
   • contradiction: a statement in the draft that directly contradicts a judgment bullet or the step's spec,
   • altered citation: a citation from the brief reproduced incorrectly (wrong clause, article, or requirement wording),
   • invented citation or requirement: a framework citation or requirement asserted in the draft that the brief does not contain.

Absence is never a violation of the global constraints. The brief routes content across steps; a stakeholder-email step correctly omits the clause-level citations that belong to the policy step. The verifier does not judge style, tone, length, or overall quality. A plain draft that is faithful to its step passes.

This shape was forced by measurement, twice. First, a live dogfood calibration on 2026-06-11, on the launch topology whose verifier was Claude Haiku 4.5 (the session's task set was not recorded in the durable log; the durable claim is the issue tally below): of 12 verifier issues raised during real use, 2 were real and 10 were over-literal flags on drafts that were in fact faithful. (This metric counts flagged issues, not clean-draft rejections.) Second, the 9-of-9 and 3-of-9 clean-draft measurements above. Both measurements condemned the wider mandate, each on its own false-positive axis; the narrow scope stayed pointed at the failure class that matters.

The narrowness is not a weakness. It is what keeps the check answerable. "Does this draft contradict the brief?" is a question whose answer lives entirely in the two documents in front of the verifier; no external facts are required to settle it. "Is this claim true?" is not that kind of question, and no prompt engineering we measured turns it into one.

What catches fabrication instead

If the blind verifier only checks fidelity, what stands between a user and a fabricated output? Three layers, each scoped to a check that needs no facts beyond its own inputs:

Citations against a closed list. The brief carries an explicit citation set. The executor's instructions say: reproduce the brief's citations exactly as written; never drop, alter, or invent framework citations. Whether a clause number belongs to a finite, known list is checkable without seeing any source. In a compliance domain, that closed world is the right target: the dangerous output is not a paraphrase, it is a citation to text the approved set does not authorize.

An integrity gate where a source does exist. On runs that performed web research, the pipeline additionally checks output URLs and prices against the approved evidence set, because there the ground truth is available and finite.

Fail-closed plumbing, with one deliberate exception. The verifier's verdict is the only machine-readable output in the pipeline, a strict JSON object with a boolean pass and an issues list that must agree with it. A verdict that cannot be parsed throws, never passes. A failed verdict triggers exactly one re-render with the issue list, then a re-verify; a second fail releases the step with a caveat flag, having been inspected twice. A verifier outage or an unparseable verdict gets one retry, then halts the run, and the in-flight step is never released. The exception matters and is worth stating exactly: fail-closed applies to inspection, not to correctness. A pass means the verifier detected no contract violation, which is not a truth guarantee from an imperfect model, and a twice-failed flagged draft did not pass anything.

The catch, live

On 2026-09-02 we ran a one-task probe through our production pipeline on our own paid account: a task we wrote ourselves, no uploaded customer material. The purpose was end-to-end proof of the four-role topology under test that day, which ran every role on GLM 5.3 (effort high on the planner, low on the executor, verifier, and recap). The task: draft a short asset register policy outline for ISO/IEC 27001:2022 Annex A, one page.

The executor's draft cited ISO/IEC 27001:2022 clauses 9.2 and 9.3, the internal audit and management review clauses. Both clauses exist in the standard; the citation is exactly what a competent-looking compliance document reaches for. The problem is contractual, not factual: the brief's approved citation set did not contain them, which makes them invented citations in the contract's sense, citations asserted in the draft that the brief does not authorize. The verifier returned a fail verdict naming the issue, the executor re-rendered with the citations brought back inside the approved set, and the second attempt passed. The run completed in 155 seconds with no caveat and no truncation, all calls on GLM 5.3.

Two properties of that catch matter more than the anecdote. First, the class: the failure that actually occurred in production was a citation reach beyond the approved set, precisely the violation the blind check covers, not a hand-planted fact. Second, the size of the check: the verifier ran at low reasoning effort and saw two documents. In the same week's eval of the topology (2026-09-01: five hard tasks covering scope discipline, audience omission, citation correctness, cross-document consistency, and unverified-evidence handling; three trials per task per arm; 30 runs total across two arms, GLM 5.3 and a Grok 4.20 proxy arm), the GLM 5.3 arm's verifier returned a parseable strict-JSON verdict in all 15 of its runs, with zero halts. Judging was independent: Claude Haiku 4.5, run through OpenRouter. The gate that caught the production failure is the most restricted model call in the pipeline.

What this does not claim

The production catch is one data point from a deliberate probe, not a rate estimate. The plant-test harness was an internal scratch file we did not keep; we stand behind its qualitative result (most planted fakes passed, recorded catch rate roughly one in four at the time) but do not publish it as a precise measurement, because the artifacts, model versions, and task set behind it were not frozen. The durable record for the widened-rule test is the 9-of-9 and 3-of-9 clean-draft measurements in the shipped contract comment, measured and recorded 2026-07-22; the record names the draft-generating families (GLM 5.2, Grok 4.20) but does not name the verifier model or task set, so we attribute only the drafts, and we have not re-run the test on other families. The July retracted-run investigation predates our current evidence discipline, so we describe it without run identifiers or model attribution. The residual risk is real and documented in the repo: on runs without web research, an executor-invented organizational fact, as distinct from an invented citation, still has no automated gate, and we have not measured how often that occurs; fixing it would mean giving the verifier the sources, a redesign, not a prompt clause. Everything here is point-in-time as of the dates stamped: contract and code as of 2026-09-16, probe 2026-09-02, eval 2026-09-01, dogfood calibration 2026-06-11, widened-rule measurement 2026-07-22.

The checklist

Before you trust, widen, or "fix" a verification gate in your own agent pipeline:

1. Decide what the verifier may see, and let that decide what it may judge. Source-blind means fidelity only: contradictions, altered citations, invented citations.
2. Absence from a reference the producer may legitimately exceed is never a violation. If you cannot prove the fact was fabricated rather than grounded in a source you did not pass, do not flag it.
3. Make honesty mechanical wherever your domain allows. Citations against a closed list, URLs and prices against an approved evidence set: these are answerable from the inputs. Open-ended truth is not.
4. Calibrate on real drafts before trusting the gate. Measure your over-literal share before widening the contract; ours was 10 of 12 issues on live use (2026-06-11). Design the contract so absence is not a violation.
5. A verdict that cannot be parsed must halt the step, never pass it. Keep one machine-readable verdict and make the pass boolean agree with the issues list.
6. One re-render on a failed verdict, then a re-verify. Release a twice-failed step only with a caveat flag, after two inspections. Never release uninspected content on any path, including verifier outages. A pass means no contract violation was detected, not that the content is true.
7. Test with the failure class your domain actually produces (for us, citations reaching beyond the approved set), not the class that is easiest to simulate (hand-planted facts). And mind where you plant them: a fake written into the verifier's ground-truth document tests nothing about fabrication.
8. When your own investigation of a suspected fabrication depends on inputs you did not check, you are the blind verifier. Retract first, measure second.

A verifier that sees less is not a weaker gate. Its verdicts stay checkable, because it judges the one contract the producer was actually given.