How we evaluate AI models for compliance: benchmarks, methods, and findings.
Behind a multi-provider gateway, the same model ID produced its first token at a 312 ms median with no reasoning output one day, and at 3,073 ms with 2,627 characters of reasoning days later. The routing flag we expected to prevent this did not.
A general-purpose moderation classifier flagged 15 of 15 messages wrong over a 17-day window in our compliance product, because our users discuss threats for a living. The fix relocated the risk rather than removing it.
On a 14-task head-to-head (2026-06-11), our single-turn baseline scored 0.984 and tied 13 of 14 tasks, so a challenger that was never worse posted a 7.1% win rate against a 60% ship gate. The gate had stopped measuring the challenger and started measuring the tasks.
On our multi-document depth eval (2026-06-04, GLM-4.7), a clause-mapping metric scored a near-perfect 1.0 while the model's per-document analysis fell to roughly a third of its standalone depth. The obvious metric was the one that lied.
