Learn · AI gateways
What is an AI gateway?
IntSen team · Updated 12 November 2026 · 7 min read
Short answer
An AI gateway is a single control point that sits between your applications or AI agents and the model providers they call. Every model request passes through it, so one place can authenticate the caller, decide which models it may use, apply safety checks and budgets, hold the provider credentials, and record what happened.
The definition
An AI gateway (also called an LLM gateway or model gateway) is a service that receives an application's model requests, applies rules to them, forwards them to the model provider, and returns the provider's response. To the application it looks like the model API itself: the application changes the address it sends requests to, and usually adds a credential and a few identifying headers.
Because every governed model call passes the same point, the gateway becomes the natural place to enforce policy and to keep evidence. Without one, each team builds its own path to the model, with its own keys, model choices, logging and cost tracking.
What an AI gateway does
The responsibilities vary between products, but most AI gateways handle some or all of the following on every request:
- Authentication: confirm which application or agent is calling, typically with OAuth client credentials or keys issued per caller.
- Authorization: check that the caller may use the requested provider, endpoint and model, for example only approved models for data of a given sensitivity.
- Credential handling: hold the provider API keys centrally and attach them to outgoing calls, so applications never see them.
- Guardrails: check prompts, responses and tool results for prompt injection, personal data, secrets or harmful content, and record, mask or block.
- Budgets and limits: track cost per request and stop or alert when a budget or per-request limit is reached.
- Records: keep a consistent log of who called which model, what was decided, what it cost and, if policy allows, what was sent and returned.
- Routing: send the request to the right provider or model version, and in some products retry or fall back.
AI gateway vs. API gateway
A general API gateway manages traffic into your own APIs: routing, rate limiting, authentication and TLS. An AI gateway manages traffic out to model providers, and it has to understand the content of model calls. Rate limits become token budgets, request logging becomes prompt and response capture with retention rules, and security checks look at the language inside the request rather than only its headers.
| API gateway | AI gateway | |
|---|---|---|
| Typical direction | Clients calling your APIs | Your apps and agents calling model providers |
| Unit of control | Requests and routes | Model calls, tokens, and agent turns |
| Policy questions | Is this client allowed to call this route? | May this agent use this model for this data? Is the prompt safe? Is there budget left? |
| Cost tracking | Usually not | Cost per request, per agent and per use case |
| Content inspection | Rare | Central: injection, personal data, secrets, harmful content |
Some teams start with an API gateway plugin for model traffic. That works for routing and keys; the gaps usually appear around content checks, cost attribution and understanding what an agent did across several calls.
Why AI agents make a gateway more important
A chat feature makes one model call per user message. An agent can make many: it plans, calls tools, reads the results and calls the model again until it has an answer. Each of those calls can be governed at the gateway, and if the calls carry a shared identifier, the gateway can group them into one agent turn.
That grouping is what lets you ask questions a request log alone can't answer: what did the agent do for this user message, which tools did the model ask for, how much did the whole turn cost, and did any step trip a guardrail.
What a gateway cannot see
A gateway only sees traffic that passes through it. It cannot observe calls that bypass it, the work a tool does after the model requests it, steps an agent takes without asking the model, or what the application finally shows the user. Good AI gateways state these limits plainly, and good governance designs make bypassing the gateway hard, for example by not giving provider keys to applications at all.
When you need an AI gateway
An AI gateway starts to pay off when any of these are true:
- More than one team is building with AI models, and provider keys are spreading across services.
- You need to show which models are approved for which use cases and data, and prove it is enforced.
- Agents are moving into production and you need to explain what they did, turn by turn.
- Model spend is growing and nobody can attribute it to an agent, product or team.
How IntSen does it
The IntSen AI Gateway
IntSen is an enterprise AI governance platform built around an AI gateway. Applications and agents keep running where they run today; they point their model client at the IntSen AI Gateway, authenticate with OAuth client credentials, and add headers that name the registered use case, the agent and the turn.
Before any call reaches the provider, the gateway checks the registration, client, provider, endpoint and model, including whether the model is approved for the registration's data class. Provider keys stay with the gateway. A refused call returns a clear code such as model_not_allowed or task_budget_exceeded.
In the first release IntSen supports OpenAI (chat completions, the Responses API, embeddings and streaming). Azure OpenAI and Anthropic are on the roadmap.
Common questions
Is an AI gateway the same as an LLM gateway?
Yes, the terms are used interchangeably. Both describe a control point between applications and large language model providers. Some people use 'AI gateway' when the gateway also governs agents and non-text models.
Does an AI gateway add latency?
Every extra network hop adds some time, and checks such as guardrails add more. How much depends on where the gateway runs and which checks are switched on, so measure it on your own traffic during an evaluation.
Do I need an SDK to use an AI gateway?
Usually not. Most AI gateways accept the provider's own API format, so you change the base URL in your existing model client and add a credential and headers.
Can an AI gateway stop prompt injection?
It can detect many prompt-injection attempts and block or record them, including in tool results. No detector catches every attack, so treat gateway guardrails as one layer of defense alongside least-privilege tools and review.