// compare
Functional AI vs an LLM gateway
Verdict
An LLM gateway sits between your code and your model providers: one API for many models, with fallbacks, spend limits, and logging on every call — while your application still runs the prompts and agents. Functional AI works one level up: it hosts the prompt, agent, or multi-agent workflow itself as an Agentic Function, scores every version against your evals before it ships, and fails over between models for you. Pick a gateway to govern the model calls of code you run; pick Functional AI if you would rather not run the agent at all.
Functional AI is in private beta. LLM gateway details reflect the published pages of LiteLLM, Portkey, Cloudflare AI Gateway, Vercel AI Gateway and OpenRouter as of 2026-09; individual products differ, and corrections are welcome.
Choose Functional AI for
- Not running the agent yourself — the prompt, agent, or multi-agent workflow is hosted as one Agentic Function you call
- Stopping a worse version before it ships, not just metering it afterwards
- Model fallback with no failover code to write
- Shipping a new prompt or agent version without redeploying your app
Choose an LLM gateway for
- Governing every model call your own code makes — one API, keys, budgets, logs
- Running the proxy inside your own network: LiteLLM and Portkey's gateway are MIT-licensed and self-hostable
- Response caching and spend limits per key, team, or project
- Starting free: open-source, free-tier, or free-plan options are available today
Side by side
| Feature | Functional AI | LLM gateway |
|---|---|---|
| What it is | Agentic Function Runtime — hosts and runs your prompt, agent, or workflow | A proxy between your code and model providers |
| Where your agent runs | In Functional AI — your app calls the Agentic Function | In your code — the gateway carries its model calls |
| Access to many models | Any model — hosted at cost, or your own provider endpoint and keys | Yes — one OpenAI-compatible API across providers |
| Fallback when a model fails | Yes — automatic, to a backup model | Yes — retries and model or provider fallbacks you configure |
| Eval gate before a change ships | Yes — every version is scored against a dataset and threshold | Not advertised by these gateways |
| Versioning and rollback | Versioned Agentic Functions — pin or roll back without redeploying | Prompt management in some (Portkey); otherwise your code |
| Caching | Not advertised | Response or semantic caching in most |
| Spend controls | Step, token, and time budgets per Agentic Function; cost on every call | Budgets or spend limits per key, team, or project |
| Self-hosting | Hosted runtime | LiteLLM and Portkey's gateway: open source; Cloudflare, Vercel, OpenRouter: managed |
| Stage and pricing | Private beta; meters steps, model tokens at cost | Open-source, free, usage-based, and enterprise tiers |
At a glance
- An LLM gateway proxies each model call your code makes; Functional AI hosts and runs the whole Agentic Function — prompt, agent, or multi-agent workflow — and your app calls that instead.
- Both give you access to many models and automatic fallback when one fails.
- Functional AI scores every new version against a dataset and quality threshold before it ships; the gateways compared here do not advertise an eval gate.
- Gateways commonly add response caching, key management, and budgets per key or team; Functional AI does not advertise caching or per-key controls.
- LiteLLM and Portkey's gateway are MIT-licensed and self-hostable; Cloudflare AI Gateway, Vercel AI Gateway, and OpenRouter are managed services.
- The gateways here can be used today; Functional AI is in private beta.
Honest pros and cons
Functional AI pros
- One call replaces an agent you would otherwise run, instrument, and fail over yourself
- A version that fails its evals never reaches users — including a cheaper model that scores worse
- Automatic model fallback on every plan, with no failover code to write
- Cost, latency, and version come back with every call
Functional AI cons
- Private beta — access is via the waitlist or the design-partner program
- Not a drop-in proxy: you move the prompt or agent into an Agentic Function rather than pointing existing calls at a new base URL
- No advertised response caching, virtual keys, or per-key rate limits
- A hosted runtime — if policy requires an open-source proxy inside your own network, a self-hosted gateway meets that today
- No published customer stories or benchmarks yet
LLM gateway pros
- Close to drop-in: an OpenAI-compatible endpoint means existing code changes little
- Mature controls on raw model calls: budgets, caching, logging, key management
- Open-source options (LiteLLM, Portkey's gateway) you can run yourself
- Free or open-source tiers to start
FAQ
- What is an LLM gateway?
- An LLM gateway is a proxy between your application and model providers. Your code sends each model call to the gateway, which forwards it through one API — usually OpenAI-compatible — and adds retries and fallbacks, spend limits, logging, and often caching along the way. Your application still owns the prompts, agents, and workflows; the gateway governs the calls they make.
- What are the main LLM gateway options?
- Common options include LiteLLM and Portkey, both with MIT-licensed gateways you can self-host, and managed gateways from Cloudflare (AI Gateway), Vercel (AI Gateway), and OpenRouter. Choose the hosting model first — self-hosted open source or managed — then the controls you need: budgets, caching, guardrails, key management. If what you actually want is for someone else to run the agent, not just its model calls, that is a runtime such as Functional AI rather than a gateway.
- Is an LLM gateway free?
- Often, at least to start. LiteLLM and Portkey's gateway are open source under the MIT licence and free to self-host. Cloudflare AI Gateway's core features are free on all plans, Vercel AI Gateway has a free tier, and OpenRouter has a free plan with free models. You still pay for model usage, and paid tiers add features or fees — check each vendor's pricing page.
- Do I need an LLM gateway if I use Functional AI?
- Not for the calls Functional AI makes. The model is part of each versioned Agentic Function, cost comes back on every call, and Functional AI falls back to a backup model itself when the primary fails — your app calls the Agentic Function, not a model provider. A gateway can still make sense for AI calls your code makes directly, and because Functional AI accepts your own provider endpoint and credentials, a gateway could in principle sit behind it.
- What is the difference between an MCP gateway and an LLM gateway?
- An LLM gateway sits in front of model providers and governs model calls. An MCP gateway sits in front of MCP servers — the tools an agent calls — and governs tool access. Some products do both: Portkey offers an MCP gateway alongside its AI gateway, and LiteLLM can proxy calls to MCP servers.
- What is the difference between an LLM gateway and an LLM router?
- A gateway gives you access to models and carries each call; a router decides which model a request should go to. The two are often used together, with the router's choice executed through the gateway. Functional AI sits above both: it hosts the agent that makes the calls, chooses its model per version through evals, and fails over at runtime. See Functional AI vs an LLM router for the routing side.
Evaluating options? Join the Functional AI waitlist or become a design partner — design partners shape the roadmap and get early access.