Publishing real evaluation numbers from a production compliance AI is rare. These are three of ours. Read them as what they are: single-run, point-in-time measurements on a specific model and request shape. Two used synthetic, anonymized fixtures; the third reports anonymized aggregate counts from production traffic. No customer content appears in any of them. Each links to the full engineering write-up.
A green metric that hid a real regression
A clause-mapping metric scored a near-perfect 1.0 while the assistant's per-document analysis fell to roughly a third of its standalone depth.
What we measured. Multi-document depth parity: whether each of seven policies gets the same analytical depth when analyzed together as when analyzed alone, mapping each against ISO/IEC 27001:2022 Annex A. Run on an open-weight model served via OpenRouter, over seven synthetic anonymized policies.
- 0.34
- content-volume parity, against a 0.6 pass gate
- 0.99
- clause-mapping row parity (averaged across both samples), the structural metric that stayed green
- 0 of 7
- documents an independent LLM judge rated undiluted
- 0.22
- per-document output-token ratio, grouped versus standalone
Scope. Single internal baseline run, 2026-06-04, on an open-weight model served via OpenRouter. Point-in-time and specific to that model and request shape; it did not reproduce when the same fixtures ran against a frontier model on a direct API. No customer data.
Read the full write-up
When a baseline is too good to measure
A single-turn baseline scored 0.984 and tied 13 of 14 tasks, so a challenger that was never worse posted a 7.1% win rate against a 60% ship gate.
What we measured. A head-to-head gate over 14 hero-flow tasks on a fully synthetic 30-person SaaS fixture with seeded defects, scored by an LLM judge (temperature 0). Baseline was a single model turn on a frontier model; challenger was a heavier multi-step pipeline.
- 0.984
- single-turn baseline mean score across 14 tasks
- 13 of 14
- tasks tied; the challenger was never worse on any task
- 7.1%
- challenger win rate, with ties counting against it, versus a 60% gate
- 0.78
- the baseline's only sub-perfect task, an over-delivery penalty
Scope. One run per system, 2026-06-11, fully synthetic fixture, no customer data. This is a statement about rubric saturation on these tasks, not a verdict of equivalence between the two systems.
Read the full write-up
A safety classifier that flagged an entire domain
A general-purpose moderation classifier flagged 15 messages as harmful over a 17-day window, and every one was a false positive, because compliance users discuss threats for a living.
What we measured. Production moderation output over a 17-day window ending 2026-04-10, in a security and compliance product whose users routinely discuss attackers, exploits, and money laundering. The fix was a two-stage, domain-aware second-pass judge.
- 15 of 15
- moderation flags that were false positives (precision zero)
- 17 days
- production window measured
- zero
- false-positive rate on those flags after the two-stage fix
- 23 of 23
- confirmed events reconciled to alert emails in the post-fix audit
Scope. One product, one domain, a small sample over a single 17-day window ending 2026-04-10. Framed as our own measurement on our own traffic with a general-purpose moderation classifier; not a general claim about any classifier's quality outside this domain. No customer data.
Read the full write-up
