KV cache for LLMs and agents definition
A KV cache is the store of attention keys and values a language model computes for tokens it has already processed, kept in memory during inference so that each new token is generated by attending over saved state instead of recomputing the entire sequence.
A KV cache stores the attention keys and values a transformer has already computed, so generating the next token costs linear work instead of reprocessing the whole prompt. It matters for agents because an agent loop re-sends a growing context on every step, and a prefix that never changes can be reused instead of recomputed. In Sanity, an agent's orienting payload can be fetched over HTTP and inlined into the system prompt, putting that content in the fixed front of the context rather than in a per-conversation tool call.
A KV cache is the set of attention keys and values that a transformer has already calculated for the tokens it has seen, held in memory so they do not have to be calculated again. In attention, every token is projected into a query, a key, and a value. Because a text-generating model only looks backward, a token's key and value never change once the token has been processed, so they can be saved and reused.
At each step of generation the model computes a query, key, and value for the newest token only, appends that key and value to the cache, and attends across the whole stored sequence. The Hugging Face Transformers documentation describes the effect plainly: without caching, the attention cost per step grows quadratically with sequence length, and with caching it grows linearly, while the memory held by the cache grows linearly as generation proceeds.
The KV cache exists only at inference time. The same documentation notes that enabling it during training "may cause unexpected errors." You will also see it written as key-value cache, kv cache, or KV-cache, and OpenAI's documentation calls the stored artifacts KV states or KV tensors. These all name the same thing.
The KV cache is the data structure; prompt caching (also called prefix caching) is a serving feature that keeps that data structure alive between separate requests so an unchanged opening stretch of the prompt can be skipped. Every transformer uses a KV cache inside a single generation whether or not prompt caching is switched on. Across requests the cache is normally discarded unless the provider or serving framework offers prefix reuse.
vLLM calls its version Automatic Prefix Caching, which "caches the KV cache of existing queries, so that a new query can directly reuse the KV cache if it shares the same prefix with one of the existing queries." OpenAI's prompt caching guide states that "the prompt cache stores key-value (KV) tensors, not the tokens themselves," and that reuse "requires the entire rendered prefix to match."
Two limits are worth knowing. Reuse is anchored at the start of the prompt and is exact, so one changed character invalidates everything after it. And it only helps the prefill phase: vLLM's documentation says prefix caching "does not reduce the time of generating new tokens." This distinction is the practical one, because you do not control the per-token KV cache, but you control the prefix entirely through how you order and stabilize your prompt.
The KV cache matters for agents because an agent runs a loop, appending each action and each observation to a context that keeps growing, while the output of each step stays short. That skews the ratio of input processing to output generation far more than a chatbot turn does, which puts most of the cost and most of the latency in the prefill phase, exactly the part a cached prefix skips.
In Context Engineering for AI Agents (18 July 2025), Yichao Ji of Manus argues that "the KV-cache hit rate is the single most important metric for a production-stage AI agent," and reports an average input-to-output token ratio of around 100:1 for Manus. That is one team's figure for one product and one practitioner's stated opinion, not an industry average, but it describes the shape of the problem well.
The arithmetic is public. OpenAI's prompt caching guide works it through for GPT-5.6 and later: at the 0.1x read rate with 30-minute writes, writing a prefix once and fully reusing it once costs 1.35x its ordinary input cost, compared with 2x for processing it twice without caching, and across ten requests, one write and nine full reads cost 2.15x compared with 10x without caching. The same page says cached input tokens are "discounted up to 95%" and that the minimum cacheable prompt length is 1,024 tokens for GPT-5.6 and later.
Keeping an agent's prompt prefix cacheable comes down to making the front of the context byte-identical from one call to the next. The Manus post sets out three practices. First, keep the prompt prefix stable, because "even a single-token difference can invalidate the cache from that token onward," and a timestamp precise to the second at the top of a system prompt is a common way to break it. Second, make the context append-only and the serialization deterministic, since many JSON libraries do not guarantee stable key ordering and can silently break reuse. Third, mark cache breakpoints explicitly where the provider supports them, at minimum including the end of the system prompt.
Tool definitions deserve their own attention. They usually sit near the front of the context, so changing the tool set mid-run invalidates the cache for everything that follows. OpenAI lists the request settings that break a cached prefix, including `model`, `tools`, `parallel_tool_calls`, `text.format`, `reasoning.effort`, and `text.verbosity`.
Anthropic's prompt caching documentation adds a timing detail that surprises people: the default cache lifetime is five minutes, with a one-hour option, and the clock runs from the start of the request that writes or reads the entry rather than from the end of its response, so a long streaming response eats into the window.
No. Agent memory is application state you deliberately persist, such as conversation summaries, user preferences, or facts carried between sessions. The KV cache is ephemeral infrastructure state that a server evicts on its own schedule, with default lifetimes measured in minutes: five minutes by default at Anthropic and 30 minutes by default at OpenAI, with longer options at each. A useful test: if your cache expires, your agent should behave identically and simply cost more. If its behavior changes, the two have been confused.
A knowledge base is a different thing again. A knowledge base plus its index decides what text goes into the prompt. The KV cache decides what it costs to process text that is already in the prompt. They are sequential rather than alternative: retrieval chooses the content, caching amortizes it. The interaction is the part worth designing around, because retrieval that injects a different passage on every call puts fresh tokens early in the prompt and destroys prefix reuse, while a stable block of reference material placed ahead of the variable turn is what prefix caching rewards.
The KV cache is also not the context window. The context window is the token budget the model can attend to in a turn, and a cached prefix still consumes it. Caching does not change what the model can see, or what the model knows. It is arithmetic reuse, not learning.
Structured content makes an agent's context prefix cacheable when the same request produces the same bytes, which is a property of how the content is stored and queried rather than of the model. Sanity is the AI Content Operating System, the intelligent backend for companies building AI content operations at scale, and the relevant capability here is agentic context storage and MCP delivery: the content an agent reads is modeled, queried, and served in a form you can pin.
Three properties do the work. Agents on a Context MCP endpoint normally call an `initial_context` tool first to orient themselves, but if you control the system prompt you can fetch the same payload over HTTP by appending `/initial-context` to the MCP URL path, concatenate it into your system prompt, and drop the tool. Sanity's documentation frames that as removing a round trip from every conversation. The caching consequence follows from what the provider documentation says gets reused: an inlined payload is a fixed prefix, and fixed prefixes are what prefix caching reuses. GROQ queries return only the fields you project, in the shape you project them, which is what lets a pipeline emit identical bytes for identical content rather than relying on a serializer's key ordering. And a query can name the perspective it runs against, so the view of content feeding your prompt is a deliberate choice rather than whatever happens to be in flight.
In Knowledge Base mode, which is an opt-in beta feature whose features and limits may change before general availability, initial context returns the outline of the Knowledge Bases the endpoint serves. That outline is a pre-generated index listing every entry path, written by a build rather than per request, which is what lets it sit in the fixed part of the prompt. The honest trade-off is in the documentation too: an inlined payload is a snapshot, so refetch it on deploy at minimum, and after any Knowledge Base rebuild. Stability is what makes a prefix cacheable, and it is also what makes a snapshot go stale.
Unlock New Possibilities with Sanity
With KV cache for LLMs and agents under your belt, it's time to see what Sanity can do for you. Explore our features and tools to take your content to the next level.
Last updated: