LiteLLM's pitch is one function call for every provider:
response = completion(model="anthropic/claude-sonnet-4", messages=[...])
response = completion(model="gemini/gemini-2.0-flash", messages=[...])
Which looks like a thin dispatch layer and is not. The interesting thing about reading it is how little of the code is dispatch and how much is per-provider correction.
Where the work actually is
Providers agree on the shape of a chat completion and disagree on almost everything underneath:
- tool calls — names, nesting, whether arguments arrive as a JSON string or an object, whether parallel calls are supported and how they are numbered
- streaming — chunk boundaries, where usage appears, whether a final chunk carries totals or you accumulate
- errors — a rate limit is a 429, a 529, a 200 with an error body, or a stream that stops
- usage — reported per request, per stream chunk, cached-token fields that exist for some providers
- system prompts — a message, a top-level parameter, or unsupported
Each of those is a small translation and there are a hundred providers. The codebase is mostly a large, well-maintained pile of specifics, and that is not a criticism — it is what the problem is. Anybody claiming an elegant abstraction over this is either supporting three providers or has moved the mess somewhere you cannot see it.
The lesson we took
We support two providers and configure them by hand. Not because we solved something LiteLLM did not — because we do not have the problem.
The temptation when building a router is to build access underneath it: as long as we are choosing between models, why not reach every model? Reading the amount of per-provider correction in LiteLLM is a good cure. That is a maintenance commitment, ongoing, tracking somebody else's API changes forever, and it is orthogonal to whether the routing decision is any good.
So providers.yaml is fourteen lines and names an environment variable per
provider. If breadth is ever needed, an access layer goes underneath, and the
selection logic does not change.
The thing worth stealing
Its error taxonomy. LiteLLM maps provider errors onto a common set —
RateLimitError, ContextWindowExceededError, AuthenticationError — and that
mapping is what makes a fallback chain possible at all.
A fallback that catches bare Exception cannot distinguish "this model is
overloaded, try the next one" from "your API key is wrong, trying the next model
will also fail" from "the context was too long, and every model in this chain
has a smaller window". Those need three different responses and one of them is
stop.
Our chain uses typed outcomes for the same reason — error, timeout,
refused are distinct in the decision record, and a context overflow is not
retried against a smaller model.
The thing not to steal
completion() accepts a model string and does everything from that. It is a
lovely developer experience and it puts the routing decision in the caller's
code, as a string, at the call site.
Which is the right place for it when there is one obvious model. It is the wrong place when the choice depends on a rule somebody has to be accountable for — because then the policy lives scattered across every call site, and answering "what routes to the frontier model" means grepping.
That is the difference in one line. LiteLLM makes the model easy to name. We are arguing the model should not be named by the caller at all.
Credit where due
If you need many providers today, use LiteLLM or something like it. It is well-maintained, widely used, and the alternative is writing the same hundred translations yourself and getting the streaming edge cases wrong for a year.
Read it before deciding to build access into your own router. It is the most efficient way to find out how much work you were about to sign up for.