CON-routing-overhead-budget: routing overhead stays under 5% of end-to-end
p95.
The number is a judgement. What is not a judgement is that there has to be one, and that it has to be a fraction of the thing being routed rather than an absolute figure in milliseconds.
The failure it prevents
A router exists to send cheap work to cheap models. Suppose it does that well and saves 40% of provider spend, while adding 150ms to every request.
For a 4-second generation, that is nothing. For a 300ms classification call — the exact kind of high-volume, low-margin traffic a router is most useful for — it is a 50% latency increase to save money on a request that was already cheap. The router has made the good case worse in order to improve the expensive one.
Expressed as a fraction, the budget scales with the work. Fast requests get a tight budget because they cannot afford much; slow ones can carry more overhead because it disappears into the generation time. That is the correct shape, and an absolute millisecond target does not have it.
What it rules out
Mostly one thing: network calls on the request path.
That is why classification is a ladder rather than a model call. The LLM rung exists for the low-confidence tail and is not allowed to become the common case — not because model classification is bad, but because it is a network round trip to a service under someone else's load, and its p95 is not a number we control.
It is also why the pool, the concepts, the policy and the providers are baked into the image rather than fetched. A routing decision reads four in-memory structures. Making any of them a lookup would be a database round trip per request to save a deploy per config change, which is the wrong side of that trade.
And it is why the decision log is an append to a file rather than an insert. The router writes one record per model attempt; making that synchronous and transactional would put a database on the path of every request in order to record something nothing reads in real time.
What it costs
Config changes are deploys. Adding a model is a row in pool.yaml and an id in
a policy row, and then a deploy — not an edit to a live table.
That is a real cost and it is also the reason routing changes are reviewable. A routing policy that can be changed without a diff is one where nobody can say what it was last Tuesday.
Measuring it honestly
The decision record carries classify_ms and latency_ms per attempt, so the
overhead is measurable against the provider call it fronts, per concept, from
data the system already writes. Not a synthetic benchmark: the actual
distribution of the actual traffic.
That matters because the budget is on p95, not the mean. A router with a 2ms median and a 400ms tail has a fine average and a bad reputation — the slow requests are the ones people remember, and they are disproportionately the ones that fell through to the expensive rung.
The number itself
Five per cent is defensible rather than derived. It is small enough that nobody notices, large enough to fit an embedding comparison and a log append, and round enough to remember.
The value of writing it down is not precision. It is that "would this fit in the budget" becomes a question with an answer, asked before the code is written rather than after somebody notices the p95 moved.