Prompt Caching
A provider-side optimization that reuses the KV cache for repeated prompt prefixes across requests, typically reducing input token cost by ~10× for reused content. Anthropic, OpenAI, and Google offer it. Order-of-magnitude cost driver for RAG and agent workloads.
Related terms
- KV Cache
A per-request cache of the key and value tensors from prior attention layers, so tokens generated earlier in a response do not need re-computation. Dominates GPU memory in long-context inference. Prompt caching at API providers is a form of shared KV-cache reuse across requests.
- Context Window
The maximum span of tokens an LLM can consider in a single forward pass. Frontier models in 2026 support 1M-2M token contexts, but effective context (measurable retrieval accuracy across the window) usually lags nominal capacity.
Related on this site
Where this fits
Prompt Caching is part of the Data & Infrastructure vocabulary used in the Generative AI Maturity Framework. See the full glossary for the complete set of 149 defined terms, or take the free maturity assessment to see where your organisation stands.