Exact-match caching on LLM responses is nearly useless — prompts contain timestamps, names, ids, and rarely repeat byte-for-byte.
Semantic caching fixes that: embed the prompt, find a stored one within a similarity threshold, return its answer. For traffic with a long tail of near-identical questions the hit rate can be enormous, and each hit is a provider call that does not happen.
It is on our roadmap, unbuilt, and the reason is not implementation difficulty.
The threshold is a correctness knob
Similarity is a number, and the cutoff decides which requests are "the same question".
At 0.95 you catch rewordings. At 0.90 you catch more, and eventually you catch two questions that differ in a way the embedding does not weight heavily and a human would consider decisive. "How do I cancel my subscription" and "how do I cancel my subscription without losing my data" are close in embedding space and have different correct answers.
The failure is silent by construction. Nobody sees a wrong cache hit — they see an answer, and it is plausible, and it is answering a question they did not ask.
Tuning this means picking a number that trades money against a class of wrong answer whose frequency you cannot easily measure, because measuring it requires knowing what the right answer was.
Staleness has no invalidation story
An HTTP cache has ETags and a max-age; a database cache has the row it
mirrors. A semantic cache over model output has no upstream to compare against.
The answer was right when it was generated and there is no event that says it
stopped being.
If a cached answer describes your pricing and your pricing changes, the cache does not know. Invalidation becomes a manual, whole-cache decision — flush everything and lose the hit rate — or a per-entry TTL that is a guess.
It cuts across the constraints
This is the one specific to how we route.
A cached answer was produced by some model, for some request. If a money-path request hits an entry generated for a non-critical one, the response came from a model the residency filter would have excluded — and the constraint that must never be broken has been broken by a cache lookup that never consulted it.
So the cache key cannot be the prompt embedding alone. It has to include the concept and the constraints that applied, which fragments the cache along the axes that matter and reduces the hit rate exactly where the requests are most expensive.
The right design, and it makes the economics much less exciting than the pitch.
Where it would work
Not everywhere is equally hazardous. content.summarize on identical documents,
classification, extraction against a fixed schema — closed-ended tasks with
verifiable output and no personalisation.
Which is a narrower band than "cache the LLM", and it is also where we would
start: per-concept opt-in, so caching is a property a concept declares rather
than a global behaviour. A concept file already carries constraints and a
verifier; cacheable: true fits beside them, and the decision is made by
whoever authored the concept rather than by an infrastructure default.
Why it is not built yet
Because the honest version needs three things we do not have: a way to measure wrong hits, a story for staleness, and enough traffic for the savings to be worth the risk. All three arrive together, and none has.
Meanwhile the classification cache — one conversation, classified once — is already there. Same idea, tiny scope, no correctness surface: it caches the routing decision rather than the answer, so a wrong hit sends a request to a different-but-eligible model instead of returning somebody else's response.
That distinction is the whole of it. Caching a decision is recoverable. Caching an answer is not.