Use your prompt cache (correctly)

Many companies have recently changed their AI adoption strategy from a tokenmaxxing to more of a tokenminning approach. With good reason: Frontier model capabilities have continued to improve rapidly, but frontier model inference remains expensive. Claude Fable 5 current API pricing is at $10 per million input tokens. Cheaper open-weight models are closing the gap to frontier models, but can still fall short on many task-specific evals, including our own.

For these reasons, closed-weight models currently account for roughly 70% of token volume on model routing platforms like OpenRouter and Vercel's AI Gateway, while accounting for a much larger fraction of total spend: 95% on AI Gateway. Open-weight model token traffic is increasing, but it remains to be seen whether the gap to frontier models will be fully closed anytime soon.

In the meantime, companies building agents accessible to large numbers of end users will continue to spend on frontier model inference. The idea that these same companies may want to think about managing token spend carefully is not exactly profound, but there are many interesting facets to doing this that are worth talking about in more detail. This includes considerations on model selection, per-task model routing and general agent harness design. In this article, I want to talk about a specific topic: prompt caching.

There are a few fundamental concepts to understand before jumping into the details of what prompt caching is and how we are building the superglue agent harness to make use of it.

1. LLM provider APIs are mostly stateless

If you have never worked with LLM provider APIs, your mental model of how these APIs manage continuous agent conversations and sessions may be incorrect. The basic interaction model with an agent is similar to having a text message conversation: You send a prompt, you receive a response, you send another prompt, you receive another response and so on.

In reality, the actual interaction model is more akin to this: You send a prompt, the agent harness constructs a prefix of tool definitions, system instructions and additional context around your prompt, and you receive a model response. On the next request, you send the entire existing conversation history to the provider again along with your next prompt. There are provider-specific exceptions to this, but many agent harnesses tend to treat LLM provider APIs in this way.

When people talk about LLM provider APIs being stateless, this usually refers to the fact that APIs don't maintain conversation state for you. This forces applications to replay the full conversation history on each request. That said, LLM provider APIs are not fully stateless: One bit of provider-managed state that can persist across requests is the prompt cache.

2. From KV caching to prompt caching

If you look under the hood of common LLM architectures and their inference services, you will quickly encounter the concept of KV caching. The basic idea is simple: LLMs generate responses in an autoregressive loop, a single token at a time. In each iteration of this loop, the LLM computes a series of intermediate representations—attention key and value matrices—that are used to generate next-token predictions. These intermediate representations are cached so that model inference services do not wastefully recompute them in every iteration of this autoregressive loop.

KV caches usually persist only for the duration of a single model invocation—that is, while the model generates one response. Prompt caching allows developers to make use of the same principle across several model invocations by allowing a previously processed prompt prefix to be reused for a provider-defined TTL. The rough idea is: If I have already run previous conversation history through the LLM, the LLM shouldn't wastefully recompute the same attention key and value matrices for the same token-sequence prefix on every new request.

Prompt cache hits speed up inference and thus result in lower costs for inference providers, which is reflected in their input-token pricing models. Cached input tokens are often billed at roughly 10% of ordinary input-token pricing, which makes prompt cache hit rates one of the biggest levers for reducing token spend. Some providers charge a premium to write a cache entry, but subsequent cache hits usually offset it quickly. This caveat mainly matters for one-shot prefixes.

The way LLM provider APIs implement this often requires an exact prefix match to get a cache hit. That means that if your harness does not send back an exact, often byte-stable replay of the previous request's conversation history (every previous user message, agent text block, tool call and tool result), you may get a cache miss and pay full input-token pricing on your next request.

This may seem like a simple requirement, but it is often surprisingly challenging to implement. A classic pitfall is to inject dynamic values (e.g. a current timestamp or a snapshot of application state) very early in the conversation, such as into the system prompt, when reconstructing the full conversation history before each request.

A more subtle pitfall is progressively loading native tools based on skill invocations or tool-search results. This can seem like a reasonable way to keep the active tool set small. However, most LLM provider APIs prepend native tool definitions and descriptions to the system prompt. Without provider-specific support, changing that tool set changes the conversation prefix on the next request and invalidates the cache from that point. Progressive disclosure through code execution and module imports, or provider-supported deferred tool schemas are ways to mitigate this.

3. Prompt cache specifics matter

Prompt cache lifecycles differ between LLM provider APIs. These differences include per-request cache-breakpoint limitations, TTL differences and cache-routing mechanics. The Anthropic API supports up to four explicit cache breakpoints. OpenAI GPT-5.6 defaults to one implicit breakpoint, also supports explicit breakpoints and permits up to four new breakpoint writes per request.

Not just prompt cache breakpoint semantics, but also cache TTL differences can harness design choices. Anthropic uses a five-minute default with an optional one-hour tier, while GPT-5.6 currently guarantees a 30-minute cache lifetime. These differences affect session-compaction strategies and require applications to define prompt cache strategies at LLM provider API level.

Prompt-cache driven session reconstruction

One of the most important superglue agent harness design goals (aside from being useful and actually fixing users' problems and all) is to maximize input-token prompt cache hit rates. The most important takeaway from the previous paragraphs is that prompt caching forces you to design your agent harness with the following condition: Once any bit of context, such as a user prompt, model response or tool-call result, enters conversation history, that session history becomes immutable and must be replayed in the original provider-native representation on each request.

superglue agents are user-scoped and single-player. Our prompt cache-breakpoint strategy lets us reuse cached prompt prefixes across different users and across different superglue organizations. A rough mental model of a superglue agent session looks as follows: A stable prefix of base tools, followed by a system prompt composed of static and dynamic sections, followed by an append-only conversation history.

The key design choice is to order system-prompt content from least dynamic to most dynamic, with cumulative cache breakpoints between layers. The Anthropic API lets callers submit a maximum of four prompt cache breakpoints per request. We set the first breakpoint after the base tools and globally stable system-prompt section. Within the provider's cache-isolation boundary, this prefix can be reused across users and superglue organizations. The second breakpoint follows organization-stable context and can be reused by users within the same organization. The third follows user-stable context and can be reused across sessions belonging to the same user.

These three cumulative prefixes consume three of the four available cache-breakpoint slots. We place the fourth breakpoint at the conversation tail, allowing the next model invocation to reuse the warm session history. Across warm sessions, our median cache-hit rate exceeds 92%.

Compacting proactively to avoid cold-start penalties

One large remaining source of prompt cache misses is cold starts. A cache can expire after five minutes of inactivity on Anthropic's default tier, while other providers and cache tiers keep entries warm longer. If a user drops off after a long session with a superglue agent but comes back after a few hours to continue that cold session, we may pay full input-token pricing for very large conversation histories.

One way to mitigate this is proactive compaction. Whenever a session and its prompt cache sit idle for a sufficient amount of time—for example, for a time approaching the prompt cache TTL—compacting context proactively can be beneficial. Viktor describes a concrete version of this pattern. Their implementation summarizes against the full warm history instead of a shorter cold slice, then schedules compaction shortly before the cache expires. Counterintuitively, sending more context can be cheaper when the full history receives cached-input pricing but the rewritten slice would be uncached. If a user never returns to the idle session, compaction was unnecessary and the worst-case scenario is that you incur pointless, one-time cached-input-token and compaction costs. If the user does return, you will have compacted the context window and will pay for a much smaller number of uncached input tokens when the user continues the session.

Doing this well depends on the likelihood of users continuing cold agent sessions, the exact timing of compaction, the compression ratio, expected future turns and acceptable information loss. This remains an active topic we are exploring at superglue.

Takeaways

• Treat the prompt cache as a core cost and design constraint in your agent harness.
• Monitor cache hit rates alongside token spend and latency.
• Keep model-visible history append-only. Rewriting prior context invalidates the reusable prefix.
• Order context from globally stable to session-specific, with cache breakpoints between reuse scopes.