AI Fundamentals
KV Cache
A per-request cache of the key and value tensors from prior attention layers, so tokens generated earlier in a response do not need re-computation. Dominates GPU memory in long-context inference. Prompt caching at API providers is a form of shared KV-cache reuse across requests.
Related terms
Related on this site
Where this fits
KV Cache is part of the AI Fundamentals vocabulary used in the Generative AI Maturity Framework. See the full glossary for the complete set of 149 defined terms, or take the free maturity assessment to see where your organisation stands.