What Is an LLM Gateway? (And Do You Need One)

Most teams end up here the same way: they integrated one model provider's SDK, then a second for a feature that needed a different model, then a third when the first one had an outage during a launch. Now there are three SDKs, three billing dashboards, three sets of retry logic, and nobody who can answer "what did we spend on LLM calls last month" without exporting three CSVs. An LLM gateway is the fix for that specific mess. This post defines the term plainly, covers what a gateway does and doesn't solve, and lays out the three types you'll actually run into when picking one.

What an LLM gateway actually is

An LLM gateway is a single API endpoint, usually one that mimics an existing wire format like OpenAI's or Anthropic's, that sits between your application and every model provider you use. Your code sends requests to the gateway's base URL instead of to OpenAI, Anthropic, or Google directly. The gateway holds the actual provider credentials, translates your request into whatever shape each backend expects, and returns a normalized response in a format your application already knows how to parse.

OpenRouter's engineering writeup on the category puts the mechanics simply:

An LLM gateway acts as a middleware layer between your application and multiple AI model providers.

This is the load-bearing idea. Once that middleware layer exists, everything else people associate with gateways (routing, failover, cost tracking, logging) becomes possible because the gateway is already positioned to see every request before it leaves and every response before it comes back. Without that single choke point, each of those features would need to be built separately into every service that calls a model.

The category splits by wire format too. Some gateways expose an OpenAI-compatible endpoint only. Others, including llm11's gateway, speak both the OpenAI and Anthropic request and response shapes, so an application built against either SDK can point its base URL at the gateway and keep the rest of its code unchanged.

What problems a gateway actually solves

Three problems come up constantly once you're calling more than one model, and a gateway addresses all three at the same layer:

• Provider fallback. When a provider has an outage, rate-limits you, or returns malformed output, the gateway can retry against a different provider without your application code knowing anything happened. This is the reason a lot of teams adopt a gateway before they care at all about cost.
• Unified billing and observability. One dashboard, one log stream, one place to see spend and latency across every model you call, instead of stitching together three providers' separate consoles. For a team of any size, this alone tends to justify the integration work.
• Avoiding N different SDKs. Every provider has its own client library, its own auth scheme, its own quirks in how streaming or tool calls are shaped. A gateway collapses that down to one integration, which matters most when you're adding your fourth or fifth provider and don't want to repeat the same wiring again.

None of these three require the gateway to understand what a "good" answer looks like. They're infrastructure problems: routing, normalization, and logging. That distinction matters for the next section.

Does a gateway check whether the answer is any good?

Not by default, and this is the most commonly missed limitation of the category. A gateway sitting in the request path can see that a response came back with a 200 status, arrived in an acceptable amount of time, and doesn't obviously look like an error. What it typically cannot tell you, on its own, is whether the model's answer is actually correct, whether it's grounded in whatever context you supplied, or whether it hallucinated a plausible-sounding fact.

That's a genuinely separate concern from gateway infrastructure, and conflating the two is how teams end up disappointed. A gateway that only routes and logs will faithfully route and log a wrong answer straight through to your user. Checking the answer itself (schema validation, groundedness checks against source material, cross-model comparison, or heavier evaluation) is a different layer of the stack, sometimes bundled into a gateway product and sometimes not. If you're evaluating a gateway specifically because you're worried about answer quality, ask directly whether it verifies output or only moves and records it.

A short taxonomy of gateway types

In practice, "LLM gateway" covers three meaningfully different products, and knowing which one you're looking at changes what you should expect from it.

Source: Braintrust, "6 best LLM gateways for developers in 2026", for LiteLLM and Portkey's model coverage figures.

A self-hosted proxy like LiteLLM is software you deploy yourself: you control routing rules, retries, and logging, and you pay providers directly with no markup, at the cost of operating one more service. Our full comparison of LiteLLM against OpenRouter goes deeper on that self-hosted-versus-hosted tradeoff specifically, and our comparison of Portkey against LiteLLM covers the other common pairing teams weigh against each other, since Portkey represents the hosted, hands-off side of that same self-hosted-versus-hosted axis. A hosted router removes the operational burden entirely: no server to patch, no scaling to think about, usually billed as a fee layered on top of provider costs. Both of those categories, as the taxonomy above shows, are primarily concerned with getting a request to the right provider and back, not with checking what came back. The third category, a hosted router with verification, adds that missing piece: a check runs against the response, and a failed check triggers an automatic escalation to a stronger model rather than shipping the answer as-is. Our dedicated llm-router page covers how that routing decision itself gets made, independent of the verification step layered on top of it.

Is an LLM gateway the same thing as an API gateway?

No, though the terms get used loosely enough that the confusion is understandable. A general API gateway (think Kong, Apigee, or a cloud provider's API management product) handles auth, rate limiting, and routing for HTTP APIs broadly, with no particular awareness of what an LLM request or response looks like. An LLM gateway is purpose-built for this one kind of traffic: it understands token counts, streaming completions, tool-call formats, and provider-specific quirks like how each vendor structures a system prompt. You can technically put a generic API gateway in front of model providers, but you'd be rebuilding, by hand, the normalization and cost-tracking logic that an LLM-specific gateway already ships with.

There's a practical reason this distinction matters beyond terminology. A generic API gateway treats a request body as an opaque blob it doesn't need to understand, which is fine for most HTTP traffic. An LLM gateway has to actually parse that body to do its job: it needs to know where the model name sits in the payload to route on it, where token usage is reported in the response to bill on it, and how a streamed response is chunked so it can pass that stream through without breaking it. That extra layer of protocol-specific parsing is what a plain API gateway would need bolted on before it could do anything an LLM gateway does out of the box.

Where llm11 fits

llm11 is a hosted router with verification: one OpenAI- and Anthropic-compatible endpoint that sends each request into a pool of candidate models, picks the cheapest one likely to handle it, and runs a check on the answer before it goes out. Schema and groundedness checks run on every request; heavier checks run on the requests that earn them. If a check fails, the request escalates once to the strongest model in the pool automatically, so a wrong answer from a cheap model doesn't just ship because nothing was watching. Billing is prepaid credits at provider cost plus a purchase fee rather than a subscription, and the pricing page has the actual numbers.

If what you need is strictly normalization and failover with no opinion on answer quality, a self-hosted proxy or a plain hosted router is a reasonable, simpler choice. If wrong answers reaching users is the failure mode you're actually trying to prevent, that's the gap a verification layer is built to close, and how it works walks through the mechanism end to end. Either way, the quickstart is the fastest way to see the request and response shape for yourself before committing to anything.

Frequently asked questions

Is an LLM gateway the same as an LLM router?

Not quite. A gateway is the layer that normalizes requests and responses across providers; a router is a decision-making component that can sit on top of a gateway and picks which model handles each request, usually to save money. Plenty of gateways include a router, but the two words describe different jobs.

Does using a gateway slow down my requests?

A well-built gateway adds a small amount of latency, typically low single-digit milliseconds for the routing decision itself, well below the variance you'd see between two different model providers on the same request. The bigger latency factor is almost always which model answers, not the gateway layer in front of it.

Can I switch providers without a gateway?

Yes, but you're rewriting the parts of your code that call the provider's SDK directly, updating auth, and reworking any provider-specific request shaping, every time. A gateway is what lets that become a base URL change instead of a code change.

Do I need a gateway if I only use one model provider?

Probably not yet. The case for a gateway strengthens once you're calling a second provider, need failover for reliability, or want spend visibility across more than one bill. A single-provider setup doesn't have the normalization problem a gateway is built to solve.

What does "OpenAI-compatible" mean for a gateway?

It means the gateway accepts requests in the same JSON shape as OpenAI's API and returns responses in that same shape, so an application built against OpenAI's SDK can point its base URL at the gateway with little to no code change. Many gateways, including llm11, support both the OpenAI and Anthropic shapes for broader compatibility.