Most LLM routers answer the same question: is this request hard enough to deserve the expensive model? Score the prompt, pick a tier, send it. It is a reasonable question and it produces measurable savings, which is why almost everyone builds it.
It was the wrong question for us, and the reason is worth more than the conclusion.
The request that broke it
A request arrives asking whether a retry path can double-charge a customer. On a difficulty axis it is unremarkable — a short question about some code, no long context, no tool use, nothing that scores as hard. A difficulty router sends it to a cheap model and is right to, by its own logic.
But we do not care how hard that question is. We care that it is about money, and that answering it means showing the model a ledger. Where the weights run is a compliance decision, and difficulty has no way to express it. There is no score you can assign that means this must not leave our infrastructure.
You can bolt the constraint on — a flag, an override, a special case — and people do. What you get is a system where the routing logic and the rules that actually bind live in different places, and the rules are the ones written in the margin.
The axis we chose instead
A request belongs to a concept: a kind of work with its own requirements.
moneypath.verify is checking a change that touches value transfer.
content.summarize is compressing a document. code.repair is fixing a
failing test.
A concept is not a difficulty band. It is a claim about what the request is, and from that everything else follows: which models are eligible, which capabilities are required, what the objective is when ranking the survivors.
concept: moneypath.verify
constraints: residency = in_infra, min_tier = high
objective: quality
candidates: [infra-large, vendor-frontier]
That residency = in_infra is not a preference the ranker weighs. It is a gate
the ranker never sees past — the candidate list it receives has already had
vendor-frontier removed, and there is no score high enough to bring it back.
What it costs
Concepts have to be authored. Somebody writes down what kinds of work exist, with three example sentences each, and that is a judgement rather than a measurement. A difficulty score needs no such meeting.
The classification also has to be cheap, because it runs on every request. Ours is a ladder — an explicit header, then a per-conversation cache, then an embedding comparison, and only then a model call for the low-confidence tail. The budget is 5% of end-to-end p95, and anything that would blow it is not allowed on the request path.
What it buys
Every routing decision becomes explainable in terms someone can argue with:
concept: moneypath.verify (embedding, confidence 1.0)
eliminated: vendor-frontier — residency
chosen: infra-large
That last line is not a log entry. It is the product. When someone asks why their request went where it did, the answer is a sentence about what the request was, not a number about how hard a scorer thought it looked.
And when the answer is refused — a critical request with no in-infrastructure model available — it is refused rather than quietly downgraded to the vendor model that ranked below it. A budget control that silently swaps in a cheaper model is an invisible quality change, and the caller cannot tell the difference between "we saved you money" and "your results got worse".
The routing policy described here is public: the concepts, the pool and the
constraints are all readable, and any single decision can be inspected without
being served at /v1/explain.