Direct LLM API vs LLM Gateway: Portkey vs Kong AI vs LiteLLM vs Custom
Direct provider APIs are the fastest way to prototype and the most expensive way to run production at scale. LLM gateways sit between your apps and providers, offering unified auth, cost attribution, rate limits, retries, and observability — at the cost of an extra hop and dependency.
Last reviewed:
What we're comparing
- Direct provider APIs
Applications call Anthropic / OpenAI / Google SDKs directly. Simplest architecture.
- LLM gateway
Middleware layer between applications and providers. Handles auth, rate limiting, cost attribution, retries, routing, observability. Examples: Portkey, Kong AI, LiteLLM, Cloudflare AI Gateway, custom.
Side-by-side
| Attribute | Direct provider APIs | LLM gateway |
|---|---|---|
| Setup time | Minutes | 1-3 days for a real deployment |
| Cost attribution | App-by-app (each pays separately) | Per-team / per-user / per-project centrally |
| Retry / fallback logic | Per-app implementation | Centralized with configurable policy |
| Provider fallback | App-side switching required | Fall back automatically if provider unavailable |
| Model routing | Per-app rules | Centralized routing policy across all apps |
| Observability | Per-app instrumentation | Unified dashboard for all LLM traffic |
| Rate limiting per tenant | App-implemented (often skipped) | Native, enforced centrally |
| Best for | Prototypes, single-app deployments, MVP | >2 production apps, multi-team GenAI programs, cost governance at scale |
When to use which
- Use Direct provider APIs
You are in early prototype phase or running a single application. The gateway overhead is not yet justified.
- Use LLM gateway
You have >2 production apps, cross-team cost attribution is required, provider failover is a business requirement, or you want a single place to enforce policy.
FAQs
Portkey vs LiteLLM vs Kong AI — how to choose?
Portkey: SaaS-first, strong dashboard + observability, fastest to deploy. LiteLLM: open-source, deploy anywhere, minimal opinion, easy to extend. Kong AI: enterprise API-management heritage, best if you already run Kong. Cloudflare AI Gateway: fits if you already run Cloudflare Workers.
What is the latency overhead?
Typically 5-30ms per request depending on gateway location and features enabled. Negligible for interactive UX. If latency matters extremely (voice UX with sub-500ms budget), colocate the gateway with your app.
Can we build our own gateway?
Yes but consider the roadmap. Building it is a weekend; maintaining it (adding new providers, keeping SDK adapters current, adding features that vendors add regularly) is a small ongoing team. Buy for stable production; build if the LLM stack is your core product.