GRC Bench

Why

A test for the work itself

We are building GRC Bench first to assess the models we consider for our own product. We publish it so the test can be inspected, criticised and reused. There are no scores on this page.

Our conflict of interest: we sell an AI product for this work, it will be an entry, and we wrote this benchmark.

Tasks

Four kinds of task

Covers GDPR, NIS 2, DORA, the EU AI Act, the Cyber Resilience Act, HIPAA, NIST SP 800-171, ISO 27001 and SOC 2. Public-law and NIST answers cite the official text.

Scoring

Quality and critical failures, reported separately

Each task has pass or fail criteria, and a task score is the share passed. Critical failures, such as a wrong deadline or an invented provision, are reported as their own rate and never netted against quality. The protocol calls for two judges from different model families, and both must pass every criterion.

Commitments

The rules we hold ourselves to

• Tasks, reference answers, scoring rules, judge prompts and the scorer are public.
• In official rounds, our product will be tested only through its public API, like any other entry.
• We will not tune framework knowledge on benchmark tasks. This is an internal rule, not a technical wall.
• Every official round is pre-registered and publishes every entry, including ours when we lose.
• We will also use GRC Bench privately to choose our own models. Those runs will not be official scores and we will not quote them publicly.
• Every correction goes in a public errata log with a new version. Official results must not compare versions.

Limits

What this preview does not show

• No scored round has been run. Nobody yet knows whether these tasks separate strong models from weak ones.
• Models wrote the tasks and answer keys. A second model checked them. That is not outside expert review.
• ISO/IEC 27001:2022 and SOC 2 are copyrighted, so tasks never ask what a control says. They use fictional situations scored on good practice. Those keys were written by a model and checked by a second model, with no official text to check against.
• All tasks are public, so anyone can tune to them, including us. Public task scores are not a ranking.

Next

What comes before v1.0

• A calibrated scored round
• Named, pinned judges with measured agreement
• Expert review of every answer key and an independent practitioner sample
• About 120 tasks and a sealed set, written apart from the ISMS Copilot team

How to run it

Run it on the AI you use

Every run is unofficial and needs two judge models from other model families than the one tested, each with its own API key. A full run makes 144 judge calls.

1. 1. Answer the 60 questions in questions.jsonl, which has the solutions removed and neutral ids. Do not open the tasks folder first.
2. 2. Write the answers to answers.jsonl, one line per question.
3. 3. Set the two judges' API keys as environment variables in your own shell, and give the agent only their endpoints and model names. It runs judge-check, then score.
4. 4. Read score.json: pass rate, critical failures, and every criterion that lost points.

Ask your AI agent: give it this page and say "Run GRC Bench on yourself and tell me where you lose points." It follows the agent steps in the repository. To run a model through its API instead, use the run kit in the README.

Use it

Run it, or tell us what is wrong

The kit runs against an OpenAI-compatible chat completions endpoint with your own key. If a task or answer key is wrong, open an issue citing the official source. Corrections are logged in public.

Published by Better ISMS EURL, maker of ISMS Copilot. Reference answers are provisional benchmark keys, not legal advice or operational instructions. Verify current authoritative sources before relying on them. ISO and SOC 2 are marks of their owners, used only to name the standard and examination. Not affiliated with or endorsed by ISO, IEC, AICPA, NIST or any regulator or standards body. Companies are fictional; any resemblance to a real company is unintended.