vLLM
A high-throughput open-source LLM inference and serving stack developed at UC Berkeley. Introduced PagedAttention (KV cache paging); the reference implementation for efficient open-weights model serving. Widely used in enterprise self-hosted deployments.
Related terms
- KV Cache
A per-request cache of the key and value tensors from prior attention layers, so tokens generated earlier in a response do not need re-computation. Dominates GPU memory in long-context inference. Prompt caching at API providers is a form of shared KV-cache reuse across requests.
- Speculative Decoding
An inference-time optimization that uses a fast draft model to propose tokens which the target model then verifies in parallel, reducing latency without changing output quality. Ships in vLLM, TensorRT-LLM, and most frontier provider stacks.
Related on this site
Where this fits
vLLM is part of the Data & Infrastructure vocabulary used in the Generative AI Maturity Framework. See the full glossary for the complete set of 149 defined terms, or take the free maturity assessment to see where your organisation stands.