Routing papers report numbers like 90% of GPT-4 quality at 40% of the cost. Those results are usually reproducible and honestly obtained. They are also measured on a distribution that has three properties your traffic almost certainly lacks.
What the benchmark assumes
Quality is a scalar. A response is scored, higher is better, and two responses are comparable on one axis. That works for MMLU and MT-Bench. It does not describe a system where a technically-worse answer from a compliant model is correct and a better answer from an excluded one is a violation.
Requests are independent and identically distributed. Benchmark items are drawn from one pool. Real traffic is bimodal or worse — a long tail of cheap repetitive work and a thin band of expensive consequential work, and the thin band is where the routing decision matters and where it is least represented in any sample.
Cost is the only constraint. No residency, no context floors, no capability requirements, no per-account budget. Just price against quality.
None of those are flaws in the papers. They are what makes a benchmark a benchmark — you cannot measure across systems without fixing the axis. But it means the headline transfers to your system only if your system has the same shape, and yours has constraints or you would not be reading about routers.
The specific way it misleads
A router that scores well on a difficulty benchmark is one that sends easy work to cheap models accurately. That is genuinely valuable.
It says nothing about the failure mode that costs the most: a correct-looking answer produced by a model that a rule should have excluded. That failure does not lower any quality score. It does not appear as an error. On a benchmark it is a success — the cheap model handled it, quality was fine, cost was low.
So the metric that most rewards a router is blind to the thing most worth preventing, which is not a small caveat.
What we measure instead
We do not publish quality comparisons, because we do not have a defensible methodology for one and neither does anybody claiming otherwise about their own product.
What is measurable from the decision log, on real traffic:
- classification agreement — how often the inferred concept matches what a reviewer assigns on a sampled set, reported with the escalation rate beside it
- routing overhead — added p95 against the p95 of the provider call it fronts, under 5%
- decision coverage — the share of routed requests that produced a record, which should be 100% and is a data-integrity check rather than a performance one
All three are about whether the system does what it says. None claims a model choice was good, because that would need a ground truth we do not have.
Where published numbers are genuinely useful
For the question they actually answer: is difficulty-based routing worth doing at all?
The answer is yes, clearly, and the papers establish it. If your traffic is homogeneous and your only constraint is cost, that literature is directly applicable and you should follow it rather than invent something.
The mistake is treating those numbers as a product comparison. They compare approaches on a fixed distribution. Two products implementing the same approach will score similarly and differ entirely in what happens when a constraint has to hold, which the benchmark cannot see.
The honest version of a claim
Ours is: routing by concept makes the decision explainable and lets a constraint gate before preference ranks, with overhead under 5%.
Every part of that is checkable against our own data, and none of it is "better than X". We have not run a head-to-head against another router, and if we did, the result would mostly measure which of us was better suited to whatever distribution we picked — which is the failure this whole post is about.