The vocabulary collision: when a safety classifier flags your whole domain

A general-purpose safety classifier carries a hidden assumption: that talk about attackers, exploits, and money laundering is rare and suspicious. Point it at a security and compliance product and that assumption inverts. Over a 17-day production window ending 2026-04-10, Mistral's mistral-moderation-latest alias (resolving at the time to the pinned mistral-moderation-2603) flagged 15 messages in our app as harmful. Every one was a false positive. Precision was zero. Not degraded, not noisy: zero out of fifteen.

That is a specific, measured result on our own traffic, and it names a failure mode that is invisible until you read the flags one by one. We call it the vocabulary collision: in a specialist domain, the classifier's danger vocabulary is the domain's working vocabulary, so the signal that is supposed to be rare becomes the base rate.

The flags

Of the 15 flags, 14 landed in the classifier's dangerous_and_criminal_content category and 1 in violence_and_threats. Reading them back, the pattern was immediate. They were all legitimate GRC discussion:

• Reverse engineering a cloud provider's services, in the context of intellectual-property obligations. ISO/IEC 27001:2022 Annex A control A.5.32 is literally "Intellectual property rights"; a user working that control has to talk about reverse engineering to reason about it.
• Risk assessment written in Spanish, where "amenazas" (threats) is the ordinary word for the thing every risk register enumerates.
• Anti-money-laundering work, where "blanqueo" (money laundering) is the subject matter, not a confession.
• Red teaming and penetration-test experience.
• DAST, dynamic scanning of an application for vulnerabilities.

None of this is criminal content. It is the daily vocabulary of infosec professionals, compliance officers, and auditors. The classifier is not defective. On general web traffic, a message about reverse engineering a vendor's systems is a plausibly weak danger signal, and a general-purpose classifier is built to treat it as one. Our users are the population for whom that same signal means the opposite. We would expect this failure shape from any general-purpose classifier used the same way, because it follows from the general-purpose framing rather than from one vendor's implementation. The mistake was ours: we treated a general prior as if it were domain-aware.

Over-blocking is not a safe default

The reflex is to treat false positives as the harmless side of the precision-recall tradeoff. Miss a real threat and you have a safety hole; over-flag and you have, at worst, some noise. In production the noise had teeth, and it is worth being concrete about the second-order costs, because they are where the real damage was:

1. Alert fatigue defeated the net. Every confirmed event emailed our security inbox. Fifteen straight false alarms trained the team to glance past moderation alerts. A safety net that everyone has learned to ignore is not a safety net.
2. A right-to-erasure problem. Flagged threads were locked against deletion so evidence could not be destroyed. That is reasonable for genuine abuse. Applied to a false positive, it meant a user could not delete their own compliance conversation, which created a GDPR Article 17 (right to erasure) risk. That right, subject to its own conditions and exceptions, covers selective deletion, not only closing an account.
3. A mislabel we could not substantiate. Benign user messages were sitting in a store whose semantics were "dangerous and criminal content." That is a claim about a person we could not stand behind.

Precision, not recall, was the metric hurting us, and its failure was not cosmetic.

Why the obvious fixes do not work

Two fixes suggest themselves and both are wrong.

Tune the classifier. You cannot. The moderation endpoint is a fixed classifier. Its API accepts a model identifier and an input string and returns category scores. There is no system prompt, no instruction field, no place to tell it "this is a compliance product." A fixed classifier gives you a dial with no numbers on it.

Drop the noisy categories. Disabling dangerous_and_criminal_content and violence_and_threats would erase every false positive we had seen. It would also erase the only categories under which a genuine threat in our domain would ever surface. That is not calibration, it is turning the net off and calling the silence success.

The two-stage fix

Keep the fast classifier as stage one; it still runs on every message. When it flags something, hand the message to a second-stage judge: a small chat model (Mistral Small) given a system prompt that describes the legitimate infosec and compliance topics our users actually discuss. The judge returns one of two verdicts, CONFIRMED or FALSE_POSITIVE.

Only CONFIRMED events flow to the alert-and-lock path. FALSE_POSITIVE events go to a separate suppressed store, retained for audit and drift detection but with zero user-facing consequence: no email, no deletion lock, no criminal label. The judge fires on roughly one flagged message a day, so the added cost is a rounding error. And it fails closed: if the judge errors, times out, or returns anything that is not a clean verdict, the message is treated as CONFIRMED. The worst case is the old noisy baseline, never a silently dropped threat.

Deployed 2026-04-10, this took our false-positive rate on those flags from 15 of 15 to zero without disabling a single category.

The catch: we relocated the risk, we did not remove it

Here is the part worth taking away, because it generalizes past moderation. Adding a lenient second stage does not delete risk. It moves it, from a failure you can see to a failure you cannot.

Before, the failure mode was over-blocking, and it was loud: every false positive rang the alert bell. After, the dominant failure mode is the judge being too lenient and quietly waving through something it should have confirmed. That failure is silent by construction. It writes to a store nobody is emailed about.

We got a live demonstration of how disorienting the silent regime is. On 2026-04-14, four days after the fix, the moderation alert emails stopped completely. From the outside, a sudden zero is indistinguishable from a broken pipeline: the classifier failing open, an outage, a dropped trigger. It took reading the underlying data to tell "healthy and quiet" apart from "silently dark." Once we had read access to the suppressed store, we audited every Mistral Small verdict it had accumulated since the 2026-04-10 deployment: 40 suppressed rows, and by our reading none was a real threat wrongly dismissed. The confirmed path reconciled exactly over the same period, 23 confirmed events to 23 alert emails. The net was intact. But the only way to know that was to audit the thing that had gone quiet.

So the second stage needs its own eval, on its own schedule. The suppressed store is now spot-checked monthly for judge over-leniency, and a cheap daily count on the confirmed table watches for the pipeline going dark. The lesson is not "add an LLM judge." It is: when you fix a visible failure with a quieter component, budget for auditing the quiet, or you have traded a problem you can measure for one you cannot.

A portable checklist

If you are running a general-purpose classifier over a specialist corpus, medical, legal, security, finance, or compliance, before you trust its output:

• Check for a vocabulary collision. List your domain's normal working vocabulary. If those words overlap the classifier's danger categories, its precision on your traffic is an open question, not an assumption. Measure it.
• Read the flags, do not just count them. A precision number hides the pattern; the individual false positives reveal it. Fifteen flags told us nothing until we read fifteen flags.
• Price over-blocking honestly. Trace what a false positive triggers downstream. Alert fatigue, a deletion lock that collides with a legal erasure right, a label you cannot substantiate. Over-blocking is rarely free.
• If the classifier is fixed, wrap it, do not fight it. When you cannot instruct the model, add a domain-aware stage around it rather than disabling categories wholesale.
• Fail closed on the wrapper. A judge error should escalate, never suppress, so a wrapper outage cannot silently open the net.
• Give the quiet stage its own audit. Any component that suppresses silently needs a scheduled review of what it suppressed, plus a liveness check that distinguishes "healthy and quiet" from "dark."

Limits

This is one product, one domain, and a small sample: 15 flags over a single 17-day window ending 2026-04-10, plus 40 suppressed rows in the weeks after. It is a point-in-time result on the mistral-moderation-latest alias as it resolved during that window, framed as our own measurement on our own traffic. It is not a general claim about that classifier's quality outside our domain. The judge's own error rate is bounded by the fail-closed default but is not zero, which is exactly why it is audited rather than trusted. What we are confident in is the mechanism: when the danger vocabulary and the domain vocabulary are the same words, a general-purpose classifier's precision is a measurement you must take, not a property you can assume.