Back to Posts / Prefix Caching

Cache hit rate dropping by 20% doubles your agent's bills

• By Max Trivedi

TL;DR

  • • We posit that in the typical long agentic loops, Cache hit rate and Cached input pricing are the most dominant yet ignored cost factors.
  • • The agent loop simulation below lets you estimate costs by tweaking different parameters. This affects both API ($$) and subscriptions (usage limit).
  • • The headline claim is using the standard 1x/10x ratio between uncached read vs cached read cost. This gets even more extreme as cached reads get cheaper (e.g. Deepseek or Xiaomi pricing)
  • • Most providers on OpenRouter cache well below 50%.

Why is prefix caching necessary

Note: While this is an extensive and interesting topic, we will only touch the relevant aspect of prefix caching. Worth looking it up to understand technical details.

At any given point, the LLM you're talking to does one thing: given all of the tokens that have been seen so far and in the exact sequence they appeared, what is the next token I should generate? Once the new token is generated, it gets appended to the context and the model repeatedly asks the same question until generation finishes. This process is known as the Generation Loop and every single token generation is one Forward Pass. LLMs being sensitive to the order of input is a well recognized phenomenon (reference).

In a typical agentic loop, you have a system prompt, tool call definitions, a user prompt, tool calls themselves, tool call outputs and model reasoning tokens. Something like: 

SystemPrompt
Tool Defs
User Prompt
Model Reasoning
Model Response
Tool Call 1 (model decided to emit this tool call based on the exact sequence
              in the context window up until this point)
...
Tool Call 10 (same)
...
More reasoning
Tool Call 75 (same)
...
...
Task Completion

The process above takes many API calls and typically grows the context window into tens of thousands or even low hundreds of thousands of tokens. At every turn, the agent sends the complete history of the conversation to the LLM.

And on the LLM side, it is both expensive and unnecessary to do a full forward pass at every turn, so the infra provider typically caches what is computed so far and only does the generation loop on the remaining content. This cache is known as a Prefix Cache. In the most naive form, prefix caching is simply doing a look up of the incoming full conversation against the server-side KV cache, starting from the very first token (hence "prefix" caching), to see how far it can match an available cache.

How much Caching affects your costs/usage limits?

The chart below simulates the actual costs. Each turn appends 200–3000 random input tokens, the model responds with ~200 output + ~100 thinking tokens, and a cache hit is rolled probabilistically. Results are averaged over multiple runs. Adjust the parameters and watch the chart respond. 100% caching is obviously impossible, we use 96% as gold standard.

Parameters

Tuning

How do I know my live caching stats?

You can look into the responses from your provider to check what is the cached reads vs full reads on each response. Alternatively, Dirac continuously displays the weighted averaged cache hit rates throughout the agentic loop.

Cache hit rates update

Dirac now displays live cache hit rates on all sessions.

Cache hit rates full view

Why do providers not fix their caching?

Honestly, I don't know. It is expensive for both the user, the provider, the environment and a straight downgrade on literally everything to have poor caching. As I noted in a previous post, a majority of the providers on OpenRouter have well below 50% caching.

The most generous explanation is probably memory pressure. Caches take up too much memory. Less generous is bad configuration/infra. Even less generous and more Machiavellian is the reported numbers being different from actual numbers and providers pocketing the difference (since a user has no way to tell and must rely on the token consumption stats in the response).

A few tips if you are writing an agent

  • → Keep the prefix stable: in your agent's "rebuild conversation" step, make sure nothing is sneaking in that would break the cache.
  • → No random or variable elements such as timestamps early in the system prompt - this benefits your next loop by keeping system prompt and tool defs cached across conversations.
  • → Premature compression of the conversation, while tempting, will certainly break the cache and you might end up paying effectively more.
All Posts