A cascade answers with a cheap model, verifies, and escalates on failure.
The arithmetic is appealing. If the cheap model is right 70% of the time and costs a tenth as much, then seven in ten requests cost 10% and three cost 110%, which is a 40% saving. Better than any prediction-based routing, because deciding after an attempt beats guessing before one.
It is on our roadmap as cascade-executor, still idea, and the reason is the
word "verifies".
The verifier is the whole feature
Everything rests on checking an answer more cheaply than producing it. Where that holds, cascading is close to free money. Where it does not, you have paid for two attempts and still cannot tell if either is right.
Cheap, trustworthy verifiers exist in a narrow band:
- tests that run. A code repair either passes the suite or does not. Binary, fast, and not a judgement.
- schemas that validate. Extraction against a JSON schema is checkable.
- constraints that hold. Debits equal credits. Totals reconcile.
Outside that band, the usual proposal is an LLM judge — and now the cascade depends on a model's opinion about a model's answer, which is a second thing that can be wrong in a correlated way. Cheap models and cheap judges tend to be wrong about the same inputs.
So the verifier belongs to the concept
Which is why, in our design, a cascade is not a router-wide setting. A concept declares a verifier or it does not, and only concepts that do are eligible:
code.repair:
needs: [tool_use, long_context]
verifier: pytest
code.repair has a real verifier, so a cheaper first attempt is defensible —
the acceptance criteria say exactly that. spec.disambiguate does not, and
never will, because there is no cheap oracle for whether an ambiguity was
resolved well.
That makes cascading a property somebody authors alongside the constraints, rather than an optimisation applied globally by whoever is looking at the bill.
What it costs when it misses
Worst-case latency roughly doubles. For interactive work that can be worse than the money saved — a p95 that doubles on the 30% of requests that escalate is visible to users in a way that a smaller invoice is not.
And the cost is not simply 110%: both attempts are billable, both are logged, and both are metered. Our fallback rule already applies — where a request falls back to a second model, both attempts are billable, because both were made — and a cascade makes that the common case rather than the exception.
Which means a cascade that misfires looks like a cost increase with no obvious cause, unless the decision log distinguishes an escalation from an ordinary fallback. It does, but only because that field exists for a different reason.
The honest summary
Cascading is one of the few LLM optimisations with genuinely good economics, and it is applicable to less traffic than the arithmetic suggests. The bound is not technical: it is how much of your work has a verifier that is cheap and trustworthy.
For most products that is a subset — the code paths, the extraction, the structured tasks. Applying it beyond that subset means trusting a judge, and at that point you are not saving money, you are moving the uncertainty somewhere harder to see.
The version worth building is the narrow one, declared per concept, on work that already had a way to check itself.