Switching models mid-conversation is unpleasant. Style shifts, formatting changes, and the model has no memory of the reasoning it did not do. So a conversation keeps the model it started on.
The interesting part is the word keeps, which sounds like a promise and is deliberately not one.
What could go wrong
A conversation starts as an ordinary question. It gets general.assist,
ranked on cost, and lands on vendor-fast. Fine.
Twelve turns later the topic has moved, and the caller marks a request critical
— it is now about a refund path. Residency pins to in_infra.
A naive stickiness rule says: this conversation is on vendor-fast, keep it
there. That is the rule working exactly as written, and it has just sent
money-path work to a vendor-hosted model because of a decision made twelve turns
earlier about a different question.
Where stickiness reads from
The fix is the same one that makes ranking safe: stickiness never sees the pool.
pool → hard filters → survivors → rank / stick / fall back
The sticky preference is applied to the survivor set, every turn. If the remembered model is in it, it wins. If the filters removed it, it is not there to win, and the request is routed as though there were no sticky preference at all.
So stickiness is a preference among things already allowed, which is what every preference in this system is. It cannot resurrect an excluded model because it is not looking at a list that contains one.
The guarantee, stated precisely
A conversation tends to keep one model. It does not promise to, and the cases where it will not are knowable in advance:
- the request's constraints changed — it became critical, or needs a capability the remembered model lacks
- the remembered model is no longer in the pool
- the conversation landed on a different instance
That last one is worth being blunt about. The session store is in memory, per process, and the deployment runs two replicas. A conversation can land on either and be re-ranked.
Why that is acceptable, and where it stops being
It is safe because the sticky choice is re-filtered every turn. A conversation that hops instances gets ranked again from the same survivor set with the same policy, so the worst case is a style change, not a constraint being missed.
It is a weaker guarantee than a shared store would give, and the deployment manifest says so where somebody would find it:
Session stickiness is per-instance (MemorySessionStore), so two replicas mean a conversation can land on either and be re-ranked. That is safe — the sticky choice is re-filtered every turn — but it does weaken the guarantee. Move the store to Redis before raising this.
Written next to replicas: 2, because that is where the trade is being made.
The pattern
Stickiness, fallback and ranking are three features that all want to select a model, and all three would be holes in the constraint system if they read from the pool. They do not — not because each was written carefully, but because there is one function that produces the survivor set and the three of them are downstream of it.
A property held by structure survives the person who understood it. A property held by care survives until the next contributor adds a fourth feature that also wants to select a model.