Prefill Caching
The inference optimisation of storing the KV-cache for a prompt prefix so subsequent requests that share that prefix skip recomputation. Distinct from prompt caching (a billing feature): prefill caching is what makes prompt caching possible at the infrastructure level. Governs the cost model of long-context and multi-turn agents.
Last reviewed:
Related terms
- Prompt Caching
A provider-side optimization that reuses the KV cache for repeated prompt prefixes across requests, typically reducing input token cost by ~10× for reused content. Anthropic, OpenAI, and Google offer it. Order-of-magnitude cost driver for RAG and agent workloads.
- KV Cache
A per-request cache of the key and value tensors from prior attention layers, so tokens generated earlier in a response do not need re-computation. Dominates GPU memory in long-context inference. Prompt caching at API providers is a form of shared KV-cache reuse across requests.
- Long Context
A model's ability to accept and reason over very large inputs — hundreds of thousands or millions of tokens — in a single request. Enables patterns like whole-codebase analysis or full-document review without chunking, though effective use still depends on prompt structure and retrieval quality.
- vLLM
A high-throughput open-source LLM inference and serving stack developed at UC Berkeley. Introduced PagedAttention (KV cache paging); the reference implementation for efficient open-weights model serving. Widely used in enterprise self-hosted deployments.
Related on this site
Where this fits
Prefill Caching is part of the Infrastructure vocabulary used in the Generative AI Maturity Framework. See the full glossary for the complete set of 159 defined terms, or take the free maturity assessment to see where your organisation stands.