Data & Infrastructure

LLM Gateway

Middleware sitting between applications and LLM providers, offering unified authentication, cost tracking, rate limiting, retries, and observability. Examples: Portkey, Kong AI, LiteLLM. Reduces per-app integration burden and centralizes policy enforcement.

Related terms

AI Observability
The practice of instrumenting LLM applications to capture prompts, completions, tool calls, latencies, token counts, costs, feedback, and quality metrics — then analyzing that stream for drift, regressions, and abuse. Distinct from traditional APM; tools include Langfuse, Arize, Datadog LLM Obs, Braintrust.

Also referenced: LLM Router.

Next steps

Take the assessment— See where you stand

Where this fits

LLM Gateway is part of the Data & Infrastructure vocabulary used in the Generative AI Maturity Framework. See the full glossary for the complete set of 149 defined terms, or take the free maturity assessment to see where your organisation stands.

Related terms

Related on this site

Framework dimensions

Free tools

Whitepapers

Next steps

Where this fits