"LLM gateway" is used for at least three products that share an API shape and almost nothing else. Comparison articles put them in one table and the table is mostly noise, because the column headings assume a single job.
The three jobs:
1. Access — the proxy
Make many providers reachable through one interface.
Normalise tokenisation, streaming, tool-call formats, error taxonomies and rate-limit semantics. One key, one bill, one SDK. Add a provider and every caller gets it.
Success looks like breadth and fidelity: how many models, and how faithfully does each behave through the abstraction. The hard parts are unglamorous — provider quirks, streaming edge cases, keeping up with API changes.
OpenRouter, LiteLLM's proxy mode, and every internal "llm-gateway" service built by a platform team.
2. Operations — the control plane
Make usage of those models governable.
Keys per team, quotas, spend caps, rate limits, audit logs, cost attribution, PII redaction. The question is not which model but who is allowed to spend what, and can you prove it afterwards.
Success looks like the finance and compliance teams having answers. The hard parts are attribution and enforcement points: knowing whose budget a request belongs to, and refusing before the money is spent rather than reporting after.
Portkey, Helicone, most enterprise offerings, and the half of our own system that does credits, metering and the spend gate.
3. Selection — the router
Decide which model a given request should go to.
Given a request, choose. On difficulty, cost, latency, capability, or on what kind of work it is.
Success looks like a decision you can defend. The hard parts are classification cost and — the one that is consistently underestimated — keeping constraints intact once you add stickiness, fallback and caching, all of which want to select a model too.
This is the layer we work on.
Why the conflation causes bad decisions
Because the three have different failure modes, and a table with rows like "supports 100+ models" scores them on a dimension only one of them cares about.
An access layer fails by not supporting the model you need, or by leaking a provider quirk. Breadth is the metric.
An operations layer fails by letting spend happen it should have stopped, or by being unable to say where cost went. Enforcement and attribution are the metrics.
A selection layer fails by sending a request somewhere it should not have gone — and the expensive version of that is not a wrong answer, it is a right answer produced by a model that a rule should have excluded. Nobody notices, because the response looks fine.
That last failure mode is invisible to any benchmark of the first two, which is why "which gateway is best" has no answer.
The stack most people end up with
application
└── selection which kind of work is this, what may serve it
└── operations may this account spend, and record that it did
└── access reach the chosen provider
Three layers, often three products, and increasingly one product claiming all three.
We do selection and operations and deliberately not access — providers are configured directly, and there are five models rather than a hundred. If you need breadth, put an access layer underneath; that is a sensible architecture and not a workaround.
The question that disambiguates
When someone says they need an LLM gateway, the useful question is not which one. It is: what goes wrong today?
"I can't easily try Claude and Gemini in the same code" — access. "I don't know which team spent that $4,000" — operations. "We sent a customer's payment data to a model we shouldn't have" — selection.
Three different products. The third is the one that is quiet until it is not.