We benchmarked ISMS Copilot against the naked model and the better DIY prompt. The pre-registered verdict is a tie.
A frozen 20-task evaluation across six GLM 5.3-Flash configurations: what the knowledge module changed, where the product and best standalone prompt tied under the frozen rule, and where the knowledge-plus-documents arm outscored the product.
On this frozen 20-task compliance set, the same base model scored differently depending on the scaffold around it; the largest increase came when curated framework knowledge was injected at inference time, especially on less-known frameworks. We froze the tasks and decision rule before the run, kept the pre-registered tie verdict, and published the losses.
The setup
Six arms. Same model for every arm (GLM 5.3-Flash, identical reasoning configuration, output budget, and serving pin), same twenty questions. The only variable is what surrounds the request:
| Arm | What the model gets |
|---|---|
| Naked | The question. Nothing else. |
| DIY prompt A / B | Two independently written "senior GRC consultant" prompts, authored without seeing the tasks |
| DIY + knowledge | Prompt A plus the curated framework reference module our system injects |
| DIY + knowledge + docs | The above, plus the task's context documents and the current date |
| Full ISMS Copilot | The production assembly: persona, framework detection, injected module, date, workspace documents |
The twenty frozen tasks: sixteen standard questions across nine frameworks, four traps (a nonexistent ISO 27001 control, superseded 2013 numbering, a nonexistent GDPR article, a wrong DORA date and scope premise). Nine tasks on well-known frameworks (ISO 27001, GDPR, SOC 2), eleven on less common ones (ISO 42001, the Australian ISM, DORA, NIS 2, TISAX, Singapore's MTCS). Two tasks carry a fictional contract or design document the answer must actually cite. A judge model from a different vendor family, blind to which arm produced what, scored every answer against frozen rubrics; a deterministic checker separately verified every cited control or article identifier against the real registries.
The method was adversarially reviewed by a second AI system before the freeze, and it earned its keep: it caught answer-key errors, a checker that would have punished models for correctly refuting fake controls, and several ways the arms could have drifted. All fixed before any spend.
The pre-registered headline
The primary endpoint was frozen in advance: full product versus the better of the two DIY prompts.
| Arm | Overall | Known frameworks | Less-known frameworks |
|---|---|---|---|
| Naked | 68.0% | 66.7% | 69.0% |
| Best DIY prompt | 76.0% | 88.0% | 66.3% |
| DIY + knowledge module | 96.9% | 100.0% | 94.3% |
| Full ISMS Copilot | 92.1% | 88.9% | 94.7% |
The product scored 92.1% against 76.0%: a 16.1-point difference, 95% confidence interval [3.4, 30.2]. Our pre-registered tie band was 18.2 points, sized from how much the judge wobbles when re-judging identical answers. The difference did not clear the band. The pre-registered verdict is a tie, and that is the verdict we report. We did not loosen the band after seeing the numbers.
What the layers actually show
The knowledge module is the engine. A plain consultant prompt plus the injected module scored 96.9%, the highest of the six arms on this task set. The module is the same curated reference ISMS Copilot injects when your question names a framework, and the same one the API exposes.
The product's edge lives on less common frameworks. On ISO 27001, GDPR and SOC 2, a good DIY prompt is at parity with the product (88.0 vs 88.9). The base model knows the classics. On ISO 42001, the Australian ISM, DORA, NIS 2, TISAX and MTCS, the product scores 94.7% against 66.3% for the best DIY prompt, and the naked model scores 69.0%. Outside the trap questions, the naked model fabricated 8 control and article identifiers, including an entire invented ISO 42001 Annex A structure. The product fabricated zero.
One DIY prompt underperformed the naked arm on one task. On the ISO 42001 structure task, it scored 32% against 82% naked after confidently importing ISO 27001's Annex structure. Because the prompt changed as a whole, this run does not isolate which instruction caused the regression; adding the knowledge module raised the same task to 100%.
The losses, in plain text
The knowledge-plus-documents arm outscored the product. It scored 98.7% against 92.1%, with five task cells differing. This result shows that prompt A supplied with the same knowledge module, current date, and relevant documents outscored the production assembly on this single-question task set. The benchmark did not test the value of multi-turn state, automatic detection, workspace handling, or other workflow features.
The product failed one of the four traps. Asked to explain ISO 27001 control A.8.35, which does not exist, the product confidently described it as secure coding, which is actually A.8.28. Its injected control table in the same prompt ended at A.8.34, and both knowledge-armed DIY configurations refused the false premise. One candidate mechanism is that the production prompt's action-bias directives affected refusal behavior, but the product arm differs from those DIY arms in several components, so this benchmark cannot attribute the failure to that mechanism. It is a reproducible product defect in task t17.
Detection is name-level. One task described a personal data breach without ever naming GDPR. The product injected nothing and still answered well on base knowledge. That is the documented behavior, not a surprise: name the framework, or pin it.
What this does and does not claim
This same-base-model comparison was run on 2026-09-02 with twenty tasks, one sample per arm per task, a scripted judge, and no human calibration. On this set, scores changed in the directions reported above when the scaffolding changed; the results do not establish performance beyond this task set and run. It is not a cross-model comparison or an audit opinion, and it supports no hallucination-elimination claim. A public summary of the method, results, losses, and caveats is available on the docs page: Answer quality: the scaffold benchmark.
Related Posts
Frontier pricing is not a compliance strategy
Compliance agents are token furnaces: evidence in, framework references in, analysis out. We run the ISMS Copilot API on GLM 5.2 with curated framework knowledge injected at inference, at $2.80/$8.80 per million tokens and a $0.50/$2.00 bulk lane. Here is the pricing math against the Claude price sheet, and the evidence for why a non-frontier model holds up at compliance work.
The silent zero: when a missing judge score becomes a measurement
In our 2026-07-17 ablation of GRC document-generation strategies (GLM-5.2 and Claude Opus 4.8 as writers and as 1-10 rubric judges), pass 2 had 79 of 446 recorded judgment rows without an overall score. The aggregator counted each one as 0. The same zero-filling had manufactured a 3.25-point effect in pass 1, and when it vanished we wrote a judge-bias theory to explain why. The paired offset between the two judges on the same documents was 0.12.
Why we do not deduplicate compliance facts by embedding similarity
In our memory dedup eval, contradictions averaged 0.938 mean cosine to their nearest stored fact and true duplicates averaged 0.940, and no single threshold cleanly separated the pairs that must not merge from the ones that should. So cosine never makes the merge decision in our pipeline.