The model ID lottery: same request, different draw

Behind a multi-provider gateway, a model ID does not specify what serves your request. It names a set of deployments with different serving configurations, and unless you pin one, each request is a draw from that set. We measured this the hard way: in our evaluation of moonshotai/kimi-k2.5 through OpenRouter (186 calls, started 2026-04-22, final rounds in the days after), the same request configuration, same model ID, same provider: { sort: "latency" } hint, same ten-prompt set, produced a 312 ms median time-to-first-token with zero streamed reasoning characters in one round, 10 out of 10 calls, and 3,073 ms with a median 2,627 characters of reasoning in a later round, again 10 out of 10. A different provider served the same ID with materially different behavior, on no code change of ours. The response metadata does name the serving provider, if you think to log it; nothing flags that the behavior changed.

The sharper finding is that the routing flag we expected to prevent exactly this did not prevent it in our runs. If your product depends on reasoning depth, the model ID in your config is not what you shipped. What you shipped is a lottery ticket, and the gateway's routing controls are hints to verify, not contracts to rely on.

The same request, twice

We were benchmarking candidate models for a compliance assistant, so every configuration ran the same prompt sets: ten standard compliance prompts (explain ISO 27001 Annex A.5.1, GDPR Article 32, SOC 2 CC6.1, and so on) and five hard prompts built for multi-framework synthesis, the kind of question where reasoning depth actually shows. For every call we recorded time-to-first-token, total time, streamed reasoning characters, streamed content characters, and the provider field OpenRouter returns in each stream chunk.

With sort: "latency", early rounds routed all ten standard-set calls to one hosting provider (BaseTen): 312 ms median to first token, zero streamed reasoning characters on every call. In a later round, days apart, the same request configuration routed all ten calls to a different provider (ModelRun): 3,073 ms median, 2,627 reasoning characters median. A still later round routed back to the first. This is what sort: "latency" does by design. OpenRouter's provider routing documentation (accessed 2026-07-15) says that with an explicit sort, "load balancing will be disabled, and the router will try providers in order," ordered by measured latency. Measured latency moves, so the order moves, and with it, in our runs, whether any reasoning output appears at all.

A 10x latency swing is annoying. A behavior flipping on and off underneath a stable model ID is a different class of problem, because nothing tells you. Every response was an HTTP 200 with fluent content and plausible token accounting.

The flag we expected to catch it

We did not accept the zero-reasoning result at face value. We tried to force reasoning on the fast path with every parameter the API offers: reasoning: { enabled: true }, reasoning: { effort: "high" }, reasoning: { max_tokens: 1024 }. Zero streamed reasoning characters, every time, whenever the request was served by that provider.

Then we reached for the routing control. OpenRouter's documentation describes require_parameters this way: "When you set require_parameters to true, the request won't even be routed to that provider," referring to providers that do not support all the parameters in your request (provider routing docs, accessed 2026-07-15). We set require_parameters: true alongside reasoning: { effort: "high" }. The request routed to the same provider anyway, and the response again contained no reasoning text.

Before concluding anything, we verified our parameter format against known-good deployments, because "the model doesn't reason" and "this endpoint exposes no reasoning" look identical from the outside. Pinned to DeepInfra, the same model ID produced 3,116 reasoning characters with no flag and 3,807 with reasoning: { enabled: true }. Pinned to Novita: 3,460 characters. Those runs settle the question that matters: the model family reasons, our request syntax was valid, and the serving endpoint, not the model, was the variable.

To be precise about what this shows and what it does not: require_parameters is documented as filtering providers by parameter support, and from the responses alone we cannot tell whether our zero-reasoning results reflected the endpoint's capability metadata, provider-side parameter handling, or a serving configuration that simply exposes no reasoning text. An endpoint that exposes no reasoning may be a deliberate and legitimate serving choice; a 312 ms first token is a real product to offer, and zero streamed reasoning characters proves only that no reasoning was exposed to us, not that no computation happened. What we can say is narrower and more useful: in our April 2026 runs, declaring the reasoning parameter as required did not get us routed to an endpoint whose responses contained reasoning output. Whatever the mechanism, the practical contract we thought we had did not hold, and only output measurement told us.

Why a compliance product notices, and why yours might not

Here is the uncomfortable part, stated plainly: on our five hard prompts, the zero-reasoning configuration still looked competent. Asked to find control overlaps and a conflict across ISO 27001, GDPR Article 32, and PCI DSS for a 50-person SaaS company, it produced structured analysis with specific clause references, as recorded in our run notes: ISO 27001 Annex A controls A.9.1 through A.9.4, GDPR Art. 32(1)(a), PCI DSS requirements 3.4 and 4.1. Note the wrinkle inside the fluency: A.9.x is the numbering of the superseded ISO/IEC 27001:2013 edition; the 2022 revision reorganized Annex A. The answer read as authoritative and carried an edition-level staleness that nothing in the response flags; we caught it only by checking the numbering against the 2022 revision. That is the failure shape in miniature: fluent, structured, specific, and quietly not the behavior you had configured. If our evaluation had been "eyeball a few answers for obvious failure," the loss of reasoning output would have been invisible. It surfaced because we logged reasoning characters per call as a first-class metric, and the column went to zero.

The rest of the kimi family scan reinforced how differently each path fails. The dedicated thinking variant (moonshotai/kimi-k2-thinking) produced reasoning output on every hard prompt (median 4,103 characters) but hit 178,618 ms to first token on one deep-reasoning prompt, a three-minute stare at a spinner. Pinning kimi-k2.5 to a reasoning-capable provider (Inceptron) produced real reasoning but failed 2 of 5 hard prompts outright: the response consumed the entire 2,048-token budget on reasoning and emitted zero content. Fast without reasoning output, reasoning without bounded latency, or reasoning that starves the answer. Three failure modes across one model family, and none of them visible in the IDs. We walked away from every one of these paths for production.

For a compliance assistant the stakes are specific: the answer feeds a risk assessment or an audit response, and an answer can read as authoritative while carrying exactly the staleness an auditor probes for. "Fluent, but not the behavior you configured" is the worst failure shape in this domain, because nothing in the response marks it as a failure. OpenRouter's own engineering blog acknowledges the quality side of this variance: "Some providers serve more heavily quantized variants of a model that underperform the same model hosted elsewhere" (How OpenRouter Model Routing Works, published 2026-06-12). Our data adds a behavioral dimension to that picture: not just the same model served worse, but reasoning output present or absent behind one stable ID.

The model ID lottery

That is the named idea, and the mental model we now apply to every gateway integration: when one model ID load-balances across providers with different serving configurations, every request is a lottery draw, and the printed name on the ticket is the least informative thing about it. The model ID tells you the weights family. It does not tell you the quantization, the latency distribution, whether reasoning output will appear, or whether the parameters you marked as required will be reflected in what comes back. Those are properties of the (model, provider, day) triple, and two of the three variables are outside your config file.

The portable checklist

If you route LLM traffic through any multi-provider gateway, not just OpenRouter:

• Log behavioral outputs as first-class metrics. Reasoning characters or tokens per call, alongside latency, keyed by the serving provider the response reports. A behavior that is not measured per-response can disappear without anyone noticing. Alert on collapse in production, not just in evals.
• Treat routing and parameter flags as hints until you have confirmed them against output. Send a request that requires a capability, then check which provider served it and whether the capability actually appeared in the response.
• Before concluding "this model can't X," pin a known-good provider and A/B the flag. Our request was valid the whole time; the serving endpoint was the variable. Without the pinned verification we would have blamed the model.
• Re-sample routing across hours and days, not within one session. Ten out of ten to one provider on Tuesday is a sample, not a property. Our oscillation only appeared across rounds days apart.
• Discriminate with hard prompts and report distributions. Our ten easy prompts produced answers we could not meaningfully rank; only the five hard prompts separated answer quality across configurations, and medians hid the 178-second tail. The max column is where the unshippable configurations live.
• If deterministic behavior matters, pin the provider and re-verify on a schedule. Pinning trades away failover and sometimes latency. For a regulated domain that is often the right trade.

Limits

This is one evaluation: 186 calls, 5 to 10 prompts per configuration, on our compliance prompt set, started 2026-04-22 with final rounds in the following days (we recorded the evaluation date, not per-round timestamps). Every number here is our own measurement on our own tasks as of that window, not a claim about any vendor's current behavior, configuration, or intent. Model identifiers are the public gateway IDs as requested at the time; we did not independently record underlying model revisions. Provider deployments, OpenRouter's routing logic, and the documentation we quote (accessed 2026-07-15) may all have changed since. The require_parameters observation is one model on one provider pair; we have not swept it across the catalog. Zero streamed reasoning characters means no reasoning was exposed in the response, nothing more. And the sample sizes are decision-grade for our ship decision, directional for yours. What we are confident in is the mechanism: when serving behavior varies per provider behind a stable ID, the only capability you actually have is the one you measure on the way out.

The model ID in your config is a name for a distribution. Measure the draws.