Every policy row carries a ranked list:
spec.disambiguate:
objective: quality
candidates: [vendor-frontier, vendor-balanced, infra-large]
Somebody decided that order. It is a judgement about which model is best for that concept, made once, and it does not update when a new model version ships or when a provider quietly regresses.
Replacing it with something measured is on the roadmap as earned-rankings,
still idea. Three things have to exist first.
1. An outcome signal
Ranking on our own results needs results. The decision log has token counts,
cost, latency and an outcome — but outcome: success means the provider
returned a response, not that the response was any good.
For concepts with a verifier, a real signal exists: code.repair either passes
the suite or does not. For spec.disambiguate there is nothing, and inventing
one means an LLM judge — a model's opinion about a model's answer, wrong in ways
correlated with the thing being judged.
So earned rankings would apply first to the concepts that can be scored, which is the same narrow band that makes cascading viable. Probably the same implementation, arriving together.
2. A way to try the thing you are not choosing
A ranking learned from your own traffic has an obvious pathology: you only
observe the model you picked. If vendor-frontier ranks first, it gets the
traffic, and evidence about the others stops accumulating. The ranking becomes
self-confirming — not because the model is best, but because it was first when
the measuring started.
Fixing that means deliberately routing some fraction elsewhere to keep the
comparison alive. Which is a real cost paid on real requests, and for
moneypath.verify it is not available at all: you cannot explore across a
residency boundary, because the alternatives were eliminated before ranking.
Constrained concepts therefore cannot have earned rankings in any useful sense. The survivor set is often one model.
3. Replay, so a policy change is not an experiment on customers
replay-harness is the other idea in that horizon, and it is the prerequisite
rather than a companion.
Changing a ranking today means changing it in production and watching. That is an experiment on live traffic with no control group and no way to undo the requests already served.
What we want is: take the last N thousand decisions, re-score them under the proposed policy, and report what would have changed — which requests move, what the cost difference is, and whether any constraint outcome differs. The decision log already contains enough to do this, because it records the concept, the authored candidates, what was eliminated and why.
That is worth building before earned rankings, and possibly worth building regardless. Most routing changes are currently reasoned about rather than measured, and the log is sitting there.
What authored orders get right
Worth saying, since the direction of travel implies they are a stopgap.
An authored order is explicable. spec.disambiguate prefers the frontier
model because ambiguity work is low-volume and sets up everything downstream —
that reasoning is in a comment next to the row, and somebody can disagree with
it.
A learned order is a number with no argument. When it puts a cheaper model first, the justification is "measurements said so", and the response to disagreement is to look at the measurements — which is better epistemics and worse for the case where the measurement is subtly measuring the wrong thing.
The likely end state is both: earned rankings where there is a verifier, and authored orders elsewhere, with the policy file recording which is which. Not because the hybrid is elegant, but because the honest scope of the measurement is narrower than the set of decisions that have to be made.