Why we do not deduplicate compliance facts by embedding similarity

A general-purpose similarity score answers one question well: are these two statements about the same thing. On the model we measured, it did not answer the question a memory of compliance facts actually needs answered: do these two statements agree. We measured the gap on our own data, and it is not a margin you tune away. In our memory dedup eval, true duplicates averaged 0.940 mean cosine to their nearest stored fact, and direct contradictions averaged 0.938, and no single global threshold in that data cleanly separated the rewordings we wanted to suppress from the contradictions we could not. We call this the aboutness trap, and it is why, in our pipeline, cosine never makes the merge decision.

The failure is silent and one-directional. A missed deduplication leaves a redundant row, which is cosmetic. A false merge suppresses one of two facts that disagree, and if the fact it suppresses is "MFA is not enforced for all users," an audit-facing assistant now holds the opposite of the truth. We treat those two errors as categorically different, and the whole design falls out of that asymmetry.

The measurement

The eval runs the real backend decision path (the same evaluateCandidate the pipeline uses) over an adversarial, synthetic, PII-free dataset of GRC statement pairs. Each pair is labeled with its relation: a reworded duplicate that should be suppressed, or one of three kinds that must not merge, a distinct fact differing on one token, a direct contradiction, or a scope refinement. Run of record: 2026-06-25, mistral-embed (the v23.12 model behind the alias) for the vectors and a small Mistral judge (mistral-small-2603, temperature 0) for the decision, 261 pairs, 522 model calls, about $0.036.

The embedding study is the part worth staring at. Mean cosine of the candidate to its nearest stored fact, by relation (mistral-embed v23.12, run 2026-06-25):

relation	mean cosine	n
duplicate	0.940	10
contradiction	0.938	7
non-duplicate (one-token)	0.902	239
refinement	0.847	5

Read the top two rows. The mean contradiction sits 0.002 below the mean rewording of the same fact. The single word that inverts the truth value ("not") barely moves the vector, because both sentences are overwhelmingly about MFA enforcement, and on this model, proximity tracks that shared subject far more than it tracks the negation.

The one-token distinct facts are nearly as close. "We have completed a SOC 2 Type 1 report" versus "Type 2" is not pedantry: a Type 1 report addresses the suitability of control design as of a specified date, while a Type 2 report additionally addresses operating effectiveness throughout a specified period (AICPA, SOC 2 reporting guide). Merging them tells a prospect's security team you hold evidence you do not. In an end-to-end smoke against the real modules, "SOC 2 Type 1" and "Type 2" embedded at 0.961 cosine, above the average true duplicate in the table. That single instance is the important one: a pair that must never merge, scoring higher than the mean pair that should. The non-duplicate class as a whole averaged 0.902.

Why a single threshold does not separate them

The instinct, once you see a duplicate score high, is to set a merge threshold just below it. Our data argues against it in two ways. The class means do not separate: 0.940 for duplicates and 0.938 for contradictions are, for thresholding purposes, the same number. And we have a concrete counterexample above that number, the Type 1 versus Type 2 pair at 0.961, a must-not-merge that outscores the average true duplicate. A global cut low enough to suppress the everyday reword would, on this data, also suppress facts we measured that must be kept apart. We are not claiming to have proven that no separating boundary can exist for any dataset or any model. We are reporting that on this adversarial GRC set, with this embedding model, no single global cosine threshold separated the merge-safe pairs from the merge-fatal ones.

So embeddings do one job in our pipeline: retrieval. The cosine floor is 0.55, deliberately low, and it exists only to pick which stored fact is worth comparing the candidate against. It does not decide whether to merge. Once the nearest neighbor is retrieved, the decision is made by machinery that looks at the tokens cosine is under-weighting.

The architecture that keeps contradictions apart

Two layers, and the deterministic one is authoritative.

First, a deterministic protected-token check. We diff the candidate against its retrieved neighbor for the tokens that carry compliance meaning: negation ("not", "no longer"), quantifiers and scope ("all", "only", "administrators"), jurisdictions and regions, framework versions, SOC 2 types, numbers and units. A one-sided difference in any protected class blocks the merge, and it overrides the judge whatever the judge returns. "MFA is enforced" against "MFA is not enforced" is blocked by the negation rule regardless of how similar the two look.

Second, a conservative judge. The model must reach 0.8 confidence to suppress a candidate as a duplicate; below that it keeps both rows. It is biased to keep, because the two error directions are not symmetric: a kept duplicate is cosmetic, a merged contradiction is corruption.

The gate the system has to clear: zero false merges on the 251 must-not-merge pairs. It cleared it, zero observed false merges. As a rule of thumb, 3/251 is about 1.2%, but on a fixed, curated set that is a heuristic, not a real confidence bound and not a production guarantee. The property we actually rely on is narrower and sturdier: the deterministic check alone, with no model in the loop, blocked all 251, so the floor is held by code you can read rather than a model you have to trust.

What fooled the judge

The honest part. A companion cleanup eval two days later, 2026-06-27, surfaced one false merge an earlier version of the check missed: "SSO is available for enterprise customers" against "SSO is available for all customers." The model judged the broader statement a harmless refinement of the narrower one and merged. It is the aboutness trap one level up: the two sentences are almost entirely about SSO availability, and the judge, like the embedding, under-weighted the quantifier that changed who the fact applies to. The fix was to move a one-sided change in a scope-bearing class (quantifiers, subjects, negation, modality) into the deterministic check rather than leaving it to judgment, plus an explicit applicability-scope rule in the judge prompt as defense in depth. After the fix, that eval showed zero false merges, 18 of 18 distinct pairs kept apart, and 8 of 8 true duplicates recalled.

The pattern repeats across layers. The embedding under-weights negation and scope structurally, on this model. The judge under-weights them softly. The layer that reliably caught them in our runs is the one that names the exact tokens that flip a compliance fact, and that layer is deterministic code, not another model asked to be careful.

Limits

This is a decision-quality gate over labeled pairs, one candidate against its matched neighbor, not a large-store retrieval load test, and the contradiction and refinement classes are small (7 and 5 pairs), so their means are indicative, not tight estimates. The cosine values are specific to mistral-embed v23.12 on these GRC sentences; a different embedding model would shift them, and the negation-insensitivity we observed is specific to the model we measured, not a law of similarity embeddings. Contrastively trained models can be tuned to push contradictions apart, using them as hard negatives (SimCSE, EMNLP 2021), so the right reading is "measure your own model," not "embeddings can never do this." Duplicate recall in the dedup run was 0.5, five of ten true duplicates suppressed. That is the price we chose: the judge keeps both rows whenever it is unsure, so recall is the lever we tune while the observed false-merge count stays at zero. We would rather carry a few redundant true facts than ever merge two that disagree.

The aboutness trap, and a checklist

The idea to carry out of this: on the model we measured, a high similarity score meant two statements were about the same thing, not that they agreed, and deduplication needs agreement. When such a memory is documented information an ISMS relies on, ISO/IEC 27001:2022 Clause 7.5.3 requires it be protected from loss of integrity (ISO/IEC 27001:2022, Clause 7.5.3, 2022-10 edition); if a silent merge makes a required record inaccurate, that is a loss of integrity in exactly that sense. If you are building a memory or knowledge layer with semantic dedup, run this against it:

• Measure the distribution, do not assume it. Embed labeled duplicate, contradiction, and near-distinct pairs from your own domain, on your own model, and look at cosine by relation. If the contradiction scores overlap with the duplicate scores there, no threshold over that score will separate them.
• Use embeddings for retrieval, not for the merge decision. A low cosine floor to pick a comparison neighbor is fine. Using that same score to decide a merge is how a contradiction gets silently removed.
• Put a deterministic check in front of the merge. Enumerate the tokens that flip meaning in your domain (negation, quantifiers, scope, versions, jurisdictions, units) and let a one-sided change on any of them override the model.
• Make the judge conservative and asymmetric. Bias it to keep both rows. Decide which error is cosmetic and which is corruption, and never let the model trade the second to avoid the first.
• Gate on zero of the fatal error, tune recall separately. Hold false merges at zero as the gate; improve duplicate recall against that same gate without moving the floor.
• Put the near-misses in your dataset. The scope-broadening pair that reads like a refinement is the one that gets merged. Spelled-out numbers, lowercased acronyms, and quantifier swaps are where a judge quietly fails.

All figures above are our own measurements on our own synthetic GRC fixtures, mistral-embed v23.12 and mistral-small-2603 at temperature 0, dated 2026-06-25 and 2026-06-27. The dataset carries no customer data.