AI gateway definition
An AI gateway is a service that sits between an application and multiple AI model providers, exposing one API endpoint and one account for models from many vendors. The application authenticates to the gateway once, and the gateway holds the provider credentials, routes each request, handles failover, and logs usage.
An AI gateway is a service that sits between an application and many model providers, giving you one endpoint and one account instead of a separate integration and API key per vendor. It handles routing, failover, token-based budgets, caching, and logging on every model call. A gateway governs the call itself, not what you send: in Sanity, the AI Content Operating System, content is stored as typed fields and references to taxonomy documents, so a call can carry the exact field a question concerns and a bounded set of answers.
An AI gateway centralizes the concerns that come with calling model APIs, so an application can talk to many providers through one endpoint. The capability list is consistent across the vendors and analysts who document the category: provider failover, model routing, token-based usage tracking and budgets, caching, rate-limit handling, logging of prompts and responses, and consolidated billing.
The practical effect of consolidated billing is that a developer can call a model without holding an account with its maker. That matters most for models that are closed and hosted. Speakeasy describes the pattern as a reverse proxy for model API calls, where "the application never holds a provider API key and never needs to know which model provider is running behind the endpoint."
Two widely used examples of the category are the Vercel AI Gateway and OpenRouter. Vercel describes migration as a base URL swap, with a single gateway key replacing per-provider keys, and supports the OpenAI Chat Completions, OpenAI Responses, and Anthropic Messages request surfaces. OpenRouter is OpenAI-compatible, serves `/completions` and `/chat/completions`, and passes through provider pricing with no markup on inference, charging a fee when you purchase credits instead.
The term LLM gateway is used interchangeably with AI gateway. Tetrate draws the line this way: LLM gateway emphasizes routing to large language models specifically, while AI gateway is the broader term that also covers other AI services, agent tool use, and governance. An inference gateway, in the same vocabulary, emphasizes routing to self-hosted or in-cluster inference.
A general API gateway routes and secures your own backend services, while an AI gateway fronts someone else's model APIs and adds concerns the generic article has no concept of. The architectural idea is the same, pointed at a different problem. Speakeasy puts it plainly: the pattern is identical to an API gateway or service mesh proxy, applied to the model-call layer.
The differences are concrete. An API gateway rate-limits in requests per client or per route, while an AI gateway rate-limits in tokens, the unit providers actually bill in. An API gateway caches on exact URL match, while an AI gateway can cache semantically, on how similar a prompt is to one it has already answered. An API gateway treats a request and response as stateless, while an AI gateway has to hold a streaming connection open and stay in the loop for the whole response. Vercel sets out this comparison across the same dimensions.
A model router is a third thing, and the distinction is worth keeping straight. A router picks which model handles a request and then hands off. A gateway stays in the loop for the entire request, which is what lets it fail over mid-stream rather than only before the first byte.
A request through an AI gateway passes through a fixed lifecycle before it reaches a model provider. Speakeasy sets out six steps.
First, the application sends the call to the gateway endpoint rather than to a provider. Second, the gateway authenticates the calling service against a central credential registry, so the application never holds provider keys. Third, the gateway evaluates its routing rules to decide which provider and model should handle the request, typically on cost, capability, or availability. Fourth, it checks the rate limit and budget for that caller, returning HTTP 429 with a `Retry-After` header if the limit has been exceeded. Fifth, if the chosen provider is unavailable or errors, the gateway fails over to a configured fallback. OpenRouter documents this behavior as automatic and transparent to the caller. Sixth, the response is logged with the model used, token counts, latency, cost, and the identity of the calling service.
Step four is more work than it sounds. As Vercel notes, OpenAI alone exposes six distinct headers for rate-limit state, covering request and token limits, how much of each remains, and a separate reset timer for both. Normalizing that across providers is one of the main reasons teams reach for a gateway rather than writing retry logic per vendor.
An AI gateway buys consolidation at the price of an extra component in the critical path, and there are three trade-offs worth weighing before adopting one.
The first is the extra hop. Every call now passes through a service whose latency and outages you inherit, and Azure's architecture guidance treats the added topology complexity as a real cost of the pattern. Gateway overhead tends to be small relative to the inference time it fronts, because generating tokens dominates the round trip, but a gateway outage is a system-wide outage.
The second is a third party in the data path. Prompts and responses pass through a service you do not run, which raises questions about retention and data residency. Both named examples publish controls for this: Vercel offers zero-data-retention routing and provider allowlists, and OpenRouter's business tier offers US or EU regional routing on the same key.
The third is pricing or feature lag against the provider's own API. A gateway has to keep up with provider API changes, and provider-specific features can leak through the abstraction or arrive late. Vercel lists keeping up with provider API changes among the trade-offs of the pattern in its architecture reference.
Consolidated billing is also not universal. Both named examples support bring-your-own-key, where you keep provider accounts and the gateway supplies only the control plane. A gateway is also not the only way to get failover and logging: some teams do routing, retries, and logging in a client library with no proxy in the data path at all. The difference is whether a separate network hop exists.
An AI gateway matters most when a model you want is closed and hosted, when you are calling several providers at once, or when you need per-team spend visibility across all of them. The first case is the clearest. A model with a waitlisted, closed API is reachable through a gateway without an account with its maker, because the gateway holds the relationship and bills you.
System One models are a current example. TypeSafe AI released Jev on 15 September 2026 as the first of these: instead of generating prose, you pass state plus typed questions and get back probabilities over a fixed answer space. Direct access is closed and hosted, and the model is also listed by AI gateways, which is how most developers reach it. OpenRouter exposes it through a separate decisions endpoint rather than the chat API, a reminder that a gateway's abstraction has to stretch when a model does not look like a chat completion.
A gateway governs the call. It can route to a cheaper model, retry a failed one, and log what each team spent, but it has no view of the quality of what you sent. For typed decisions, that quality is set upstream, by whether the state you pass in is a clean bounded field or a scraped page. Content held as discrete fields and blocks rather than one blob of markup lets a call carry the exact field a question concerns, and references to taxonomy documents give the model an answer space drawn from documents that exist rather than a list hand-maintained in a prompt. Those are different layers of the same system, and a gateway does not substitute for the one beneath it.
Unlock New Possibilities with Sanity
With AI gateway under your belt, it's time to see what Sanity can do for you. Explore our features and tools to take your content to the next level.
Last updated: