KV Cache Explained: Prefill, Decode and Prompt Caching

Cached input is cheaper because the model can reuse work it already did. A KV cache keeps 2 sets of attention data, keys and values, so a large language model can continue a conversation without rebuilding every earlier token's state. That makes repeated context cheaper to process, but long conversations still occupy memory and give each new token more context to attend to.

Preserve useful prefixes, then control how much history you retain. Those are separate jobs. A prefix is the unchanged beginning of a request; caching it can reduce input processing without making the rest of an agent's work disappear.

Provider prices and cache rules below were verified against the vendors' live pricing pages and documentation on 15 October 2026. The memory calculations are derived from documented tensor shapes, not measurements of a particular GPU.

What Is KV Cache? The Work Behind a Repeated Prompt

KV cache is saved attention state. Attention is the mechanism a transformer uses to bring relevant earlier information into its current prediction. A token, a unit of text or other model input, produces numerical representations inside each attention layer. The cache retains that layer's keys and values for later use. Hugging Face's cache explanation describes this per-layer storage.

The three attention roles are easier to understand as a research library:

• A query expresses what the current position is looking for.
• A key makes an earlier position available for matching.
• A value supplies the information that a match can contribute.

Imagine preparing an indexed set of research notes before writing a report. Each new sentence can consult the prepared notes. Rebuilding the index and notes for every sentence would repeat work; keeping them occupies space. The analogy stops at the implementation: the model stores numerical arrays, not readable index cards or a summary of your documents.

The current query is compared with earlier keys, and the resulting attention weights combine their values. In a causal transformer, earlier positions cannot look into the future. Adding a later token therefore does not require changing the already computed states of earlier positions. That is what makes reuse valid. The original PagedAttention paper describes these attention computations and the generation process.

This also explains why the usual cache stores K and V rather than every old query. The next prediction needs the current query and the accumulated keys and values; the old queries have already done their job.

For a support agent, the retained states might represent its instructions, tool definitions, policy documents and conversation so far. They are specific to the model and preceding context. They are not the model's trained weights, and they are not a permanent customer-memory database.

A cached prefix is also different from a cached answer. The model still generates a fresh continuation. Asking a new question about the same document can reuse the document's prepared state without returning the previous answer.

KV Cache Explained: Prefill vs Decode

Prefill prepares the input; decode continues from it. These phases explain why a request can start faster after a cache hit while still taking time to finish.

Here is the diagram in words for a conventional autoregressive transformer:

Prefill: prompt tokens → process the prompt → retain per-layer keys and values → predict the first output token.

Decode: latest generated token + retained keys and values → process that token and attend to the context → predict the next token; append the newly computed keys and values.

Repeat decode: reuse the growing cache until generation ends.

Prefill can process the known prompt positions together. Decode advances through a continuation whose later tokens have not yet been chosen. This distinction is described in the PagedAttention paper's generation and execution flow.

The KV cache makes the next token cheaper relative to recomputing the entire prefix. It does not make the next token independent of context length. With conventional full attention, its query still interacts with the retained keys and values. More retained context means more attention data to use. Hugging Face documents this remaining per-step attention work.

Now separate two kinds of reuse. Within a response, KV caching prevents repeated calculation during generation. Across requests, prefix caching, also called prompt caching at hosted APIs, preserves eligible states so a later call can skip some prefill work. A conversation identifier alone does not establish that those states are available.

Suppose an agent reads a policy document, calls a tool, then asks the model to decide what to do with the tool result. The next request can retain the earlier instructions and document as a common prefix, with the tool result appended. If the cache is eligible and available, the provider need not prepare that common beginning again.

For an API builder, this is a reason to inspect both the delay before the first token and the time spent generating the response. A better first-token delay does not prove faster decode. vLLM's Automatic Prefix Caching documentation explicitly distinguishes prefill savings from decode time.

How Saved Work Becomes Cached Input

The discount is attached to reused input work, under a provider's billing contract. It is not a universal percentage, and it does not discount every token in a request that gets a hit.

For APIs that distinguish ordinary input, cache writes and cache reads, the input portion can be written as:

Input charge = (U × p_input + W × p_write + R × p_read) / 1,000,000

Here, U is ordinary input tokens, W is tokens billed as cache writes, and R is tokens billed as cache reads. Each p is that category's price per million tokens. The categories must be mutually exclusive: do not charge a write token again at the ordinary input rate. This is dimensional arithmetic using the OpenAI and Anthropic price categories.

Add output, applicable tool charges and any separately billed cache storage afterward. Google's explicit cache adds a retention dimension, so a write/read-only model is insufficient there. These are examples of the contracts, not a ranking of the models.

Anthropic: Mark the Reusable Prefix

Anthropic exposes prompt caching through cache_control, either at the request's top level for automatic breakpoint management or on individual content blocks. A breakpoint marks the end of the prefix to preserve. Its pricing documentation describes both approaches.

Price example, standard Claude API: Claude Sonnet 5.5 input is $2, cache reads $0.10, 5-minute writes $2.50, and 1-hour writes $4, all per million tokens. Anthropic pricing.

For this model, the minimum cacheable prefix is 512 tokens. Exact content must match through the cache boundary. Reads refresh the lifetime without another write charge, but the clock starts when the request begins, so response generation consumes part of the interval. Other Claude models have different minimums and read discounts. Anthropic prompt-caching rules.

For a document agent that returns after a long human pause, retention can decide whether the next call reads an existing prefix or writes it again. Check cache_creation_input_tokens and cache_read_input_tokens in usage; a repeated document alone is not evidence of a discounted read. Usage fields.

OpenAI: Check the Model's Write Rules

OpenAI enables prompt caching by default on supported models, and newer models support both implicit and explicit breakpoints. The behavior is model-specific. OpenAI prompt caching.

Price example, Standard short-context tier: GPT-6.1 Sol input is $2.00, cached input $0.10, and cache writes $2.50 per million tokens; the full request's input and cache rates double above 272K input tokens. OpenAI pricing, model threshold.

For GPT-5.6 and later, the minimum is 1,024 visible input tokens, and the documented cache lifetime is at least 30 minutes after the latest write or reuse. A cache-write rate replaces the ordinary input rate for those tokens; it is not an additional copy of the entire input bill. Earlier models have different write and retention rules. Current caching contract.

In the Responses API, inspect usage.input_tokens_details.cached_tokens and cache_write_tokens. Preserving a session does not guarantee a hit. Use the site's GPT-6 prompt-caching diagnostics guide when you need to investigate the request layout and reported usage.

Google Gemini: Implicit Reuse and Explicit Storage Differ

Google Gemini offers automatic implicit caching, with savings passed through on hits. For the Gemini 3.8 Flash example, the documented minimum is 4,096 input tokens. Common content at the beginning and requests close together improve the chance of reuse. Google's context-caching guide.

Price example, paid Standard Gemini Developer API: Gemini 3.8 Flash input is $0.75, cached input $0.075 per million tokens, with explicit-cache storage $0.50 per million tokens per hour, through 31 December 2026. Google pricing.

Explicit caching uses a named cached-content resource, with an expiration setting, that later generateContent requests reference. Its storage charge means retaining an unused document can still cost money. Google's caching API reference.

The API choice matters: the Interactions API supports implicit caching only. Google's current guide also excludes explicit caching for Gemini 4 Argon and onward; supported Gemini 3 and 2.5 models use generateContent for explicit caches. For the Interactions API, inspect usage.total_cached_tokens as documented in the guide. Google's API-specific rules.

For an API operator, the practical difference is whether you are observing automatic hits or deliberately paying to retain a resource. Do not build a cross-provider cache abstraction that assumes those are the same lifecycle.

KV Cache Size Grows With Retained Context

For conventional full-attention layers, raw KV memory grows linearly with retained tokens. A long conversation is a growing working set even when its beginning qualifies for a billing discount.

For equal-length sequences and uniform attention-layer geometry:

KV bytes ≈ 2 × B × T × L × H_kv × D × S

• 2: one key array and one value array.
• B: concurrent sequences, assuming their cache payloads are not shared.
• T: retained, processed tokens per sequence, including prompt and generated tokens.
• L: attention layers that retain this kind of cache.
• H_kv: key/value heads per layer.
• D: scalar values per head.
• S: bytes used to store each scalar value.

This is an element-count derivation from the documented per-layer cache shape and vLLM's key/value-head layout. It estimates the raw tensors, not total device memory.

The head count matters. Grouped-query attention, where several query heads share key/value heads, needs the KV-head count in this formula. Substituting the larger query-head count would overestimate that cache.

Keep the model and storage precision fixed and double T: the raw dense cache doubles. Double both T and the number of independent concurrent sequences: it becomes four times as large. That result is algebra, not a claim about a particular GPU's capacity or measured throughput.

For a self-hosted support service, this gives a useful capacity question: how many retained tokens are active across customers at once? The model fitting into memory is only the beginning. A server must also accommodate those customers' changing inference state.

Hugging Face's cache-strategy documentation describes the windowed-layer limit, while vLLM's prefix-cache design covers reusable physical blocks. Use the architecture's documented state layout when it differs from these conventional dense key/value arrays.

For a hosted API customer, the same growth has a different visible consequence: more tokens can be billed as reads on each turn, even with a stable hit rate. A lower unit price multiplied by an expanding input is still an expanding charge. Output generation also continues to attend to context, and rate thresholds may depend on the full input length.

KV Cache Optimization in vLLM

vLLM, an open-source LLM serving engine, gives self-hosters controls over allocation, reuse and representation. Choose the control that addresses your bottleneck before changing the deployment.

PagedAttention: Allocate Blocks as the Conversation Grows

PagedAttention lets attention operate on KV data stored in separate physical blocks. A logical sequence does not need one large contiguous reservation for its entire possible future. The original design reduces wasted allocation and enables sharing. PagedAttention paper.

The library analogy still works: give a growing research project available shelf sections, and keep a map of their order. You do not have to reserve a single enormous empty aisle before knowing how much material it will produce.

Paging does not reduce the information stored for each token by itself. It improves how that information fits into the available memory. Also, vLLM labels its Paged Attention kernel page as historical; use the idea without treating that old kernel description as today's implementation.

Prefix Caching: Skip Work Shared With an Earlier Request

Automatic Prefix Caching reuses a matching prefix's KV state. vLLM documents enable_prefix_caching=True; this is the relevant feature when repeated document questions or agent turns keep presenting the same beginning. Automatic Prefix Caching.

The cache identity includes the preceding prefix, not just the text in an isolated block. Identical paragraphs appearing after different earlier instructions therefore cannot simply be assumed interchangeable. vLLM's hash-based cache design makes that dependency explicit.

The wall is the workload: unrelated prompts provide little shared work, and long generated answers can remain dominated by decode. If you are selecting the engine itself, the vLLM vs SGLang comparison covers that broader decision.

KV Cache Quantization: Reduce Bytes per Stored Value

KV cache quantization lowers the precision used for cached numerical values. vLLM supports FP8 cache storage through kv_cache_dtype="fp8" and recommends dataset-based scale calibration for quality. The available paths depend on the attention backend. vLLM quantized-cache documentation.

Moving from 16-bit to 8-bit values halves the raw scalar payload in the formula, before scale metadata and allocation overhead. It does not halve all server memory, and it does not establish a measured speedup. Quantizing model weights is a separate choice from quantizing attention state.

For a production agent, evaluate the long tasks that matter: retaining constraints, selecting tools and recovering details from earlier messages. Smaller state is useful only if the task still succeeds. Do not infer unchanged quality from the fact that the server starts.

KV Cache Compression: Distinguish Precision From Content

Keeping less precise values and retaining fewer positions are different tradeoffs. Quantization changes how values are represented. A bounded attention window changes which positions remain available to that layer. Their failure modes should not be bundled into a generic compression claim. Hugging Face cache strategies.

A practical first move is to stop sending irrelevant tool output. That reduces future context at its source, but it is an application decision: preserve the evidence the agent needs before optimizing the retained state.

KV Cache Offloading: Trade Transfers for More Retained State

Offloading moves reusable state into another memory tier. vLLM's OffloadingConnector can retain completed blocks in CPU host memory and supported secondary tiers, then promote hits back to the GPU when needed. vLLM offloading guide.

This can help when useful prefixes outlive their GPU residency. It also introduces data movement. Compare the benefit of avoiding recomputation with the transfer work on your workload; there is no universal latency win to quote.

Under active KV pressure, vLLM can preempt requests and later recompute them. Its optimization guide recommends examining concurrency and memory allocation when preemptions become frequent. For an operator, a rising preemption count is a more actionable signal than the bare fact that GPU memory looks full.

What's Overhyped About Cache Hits

A high hit rate is not the same thing as an efficient agent. It measures reuse, while your business needs a successful task completed within an acceptable time and budget.

A conversation can reuse almost all its previous input and still grow on every turn. It can repeatedly reread irrelevant material at a discounted rate. It can also spend most of its time generating output, where prefix reuse offers no direct shortcut.

Do not preserve a bloated transcript solely to protect a cache metric. Compact or restart when the old material no longer helps, accepting that a changed prefix may need fresh processing. The decision should follow task quality and total work. The site's context-engineering guide explains how to decide what belongs in the working context.

For builders buying APIs, optimize prompt layout now if requests repeatedly share substantial content. Wait on elaborate cache management if your workload rarely repeats a prefix. For self-hosting operators, investigate allocation pressure and preemption before assuming the model needs more compute. A short, unrelated one-shot workload has little cross-request reuse to recover.

Questions About LLM KV Caches

Does ChatGPT use KV cache?

OpenAI documents KV-based prompt caching for its APIs. That establishes how its public API feature works, not every detail of ChatGPT's internal serving deployment. For a builder, rely on the selected API model's published cache contract and usage fields. OpenAI documentation.

Should I offload KV cache to GPU memory?

For GPU inference, active attention state is used on the GPU. In vLLM's offloading feature, reusable blocks move to host memory or other supported tiers and return on demand. Consider that when preserving more prefixes is worth the transfer work. vLLM offloading.

What happens when KV cache is full?

vLLM can evict unused cached blocks; under pressure from active requests it can preempt work and recompute it later. This affects latency. It is separate from exceeding a model's context-window limit. Cache management, preemption.

How to reduce KV cache?

Reduce retained context or concurrent sequences, reuse eligible shared prefixes, or evaluate lower-precision cache storage. Offloading changes where state lives rather than necessarily reducing its total size. Use task quality and observed latency to judge the result, alongside memory. Cache strategies.

Three Habits That Keep Agent Cache Hits High

Make the beginning stable and the changes append-only during an active task. Exact-prefix reuse rewards a predictable request, subject to the provider's minimum length, retention and availability rules.

1. 1

   Keep the stable system prompt first

   Keep durable instructions, tool definitions and reusable reference material before changing task data, within the API's prescribed message structure. Keep wording, tool order and serialization stable. Anthropic's hierarchy is tools, system, then messages; a changed tool set can invalidate reuse downstream. Anthropic cache structure.

2. 2

   Append history instead of rewriting it

   Preserve earlier messages and append the new user turn, assistant tool call or tool result. Avoid rebuilding prior turns with changing summaries or formatting on every call. When compaction becomes necessary, make it a deliberate boundary and establish a new stable prefix afterward. This follows the dependency between each block and its preceding tokens. vLLM prefix identity.

3. 3

   Keep timestamps out of the top

   Place the current time, request identifier and other changing metadata after the reusable material. A timestamp at the beginning can invalidate reuse for the context behind it. With explicit breakpoints, mark the end of the stable prefix, before the changing suffix. Anthropic breakpoint guidance.

The next working day, inspect a representative agent conversation at the request boundary. Compare the exact prefix across successive calls, then read the provider's reported cache writes and reads alongside total input, output and latency. Fix an unstable prefix before changing providers. If hits are already healthy and the agent is still slow, investigate retained context, output length and server contention next.

Get the newsletter for practical explanations of the systems behind AI agents.