How we evaluate AI models for compliance: benchmarks, methods, and findings.
A lab test showed our compliance pipeline's verifier ignored most of the fabrications we planted by hand. The fix we measured for it flagged 9 of 9 clean drafts, so we kept the verifier blind, and it caught, live in production, the failure class its contract actually covers.
Three pre-registered decision rules moved three chat surfaces to GLM 5.3 on 2026-09-01: the think flip (Grok 4.6 to GLM 5.3) scored 1.7 points lower on our four-task quality set and cut median first-token latency from 88.9 seconds to 4.2 on our latency probe.
A frozen 20-task evaluation across six GLM 5.3-Flash configurations: what the knowledge module changed, where the product and best standalone prompt tied under the frozen rule, and where the knowledge-plus-documents arm outscored the product.
Compliance agents are token furnaces: evidence in, framework references in, analysis out. We run the ISMS Copilot API on GLM 5.3 with curated framework knowledge injected at inference, at $2.80/$8.80 per million tokens and a $0.50/$2.00 bulk lane. Here is the pricing math against the Claude price sheet, and the evidence for why a non-frontier model holds up at compliance work.
In an August 2026 pre-registered ship-gate for our API prompt, three correct refusals to reproduce copyrighted standard text failed the IP check on one alias, because the gate scored that refusal against a 120-character usefulness floor and a compliant but useless answer is a product defect.
In our 2026-07-17 ablation of GRC document-generation strategies (GLM-5.2 and Claude Opus 4.8 as writers and as 1-10 rubric judges), pass 2 had 79 of 446 recorded judgment rows without an overall score. The aggregator counted each one as 0. The same zero-filling had manufactured a 3.25-point effect in pass 1, and when it vanished we wrote a judge-bias theory to explain why. The paired offset between the two judges on the same documents was 0.12.
In our memory dedup eval, contradictions averaged 0.938 mean cosine to their nearest stored fact and true duplicates averaged 0.940, and no single threshold cleanly separated the pairs that must not merge from the ones that should. So cosine never makes the merge decision in our pipeline.
When a production bug will not reproduce offline, you cannot validate a prompt fix by making it succeed. You validate it by checking whether your change moves the output in a consistent direction or just adds to the model's own run-to-run noise. Here is the test we ran on 126 blind A/B pairs.
Behind a multi-provider gateway, the same model ID produced its first token at a 312 ms median with no reasoning output one day, and at 3,073 ms with 2,627 characters of reasoning days later. The routing flag we expected to prevent this did not.
A general-purpose moderation classifier flagged 15 of 15 messages wrong over a 17-day window in our compliance product, because our users discuss threats for a living. The fix relocated the risk rather than removing it.
On a 14-task head-to-head (2026-06-11), our single-turn baseline scored 0.984 and tied 13 of 14 tasks, so a challenger that was never worse posted a 7.1% win rate against a 60% ship gate. The gate had stopped measuring the challenger and started measuring the tasks.
On our multi-document depth eval (2026-06-04, GLM-4.7), a clause-mapping metric scored a near-perfect 1.0 while the model's per-document analysis fell to roughly a third of its standalone depth. The obvious metric was the one that lied.
ISMS Copilot