kernel&chai
Subscribe
← Learn
AI agents & LLMs Explainer

What an AI agent really costs: the prompt-caching math

Every turn of an agent re-sends the whole conversation. Here's the formula for what that costs, and how much prompt caching takes off the bill.

on this page
  1. Every turn re-sends everything
  2. What caching changes
  3. A worked example
  4. The warm-cache assumption
  5. How to keep the cache working

An AI agent feels like a conversation. To the model, it isn’t one. The model keeps no memory between requests, so every time the agent takes a step it sends the whole story again: the instructions, the tools, the files, every earlier message and every earlier answer. Then it adds the one new thing.

That one fact decides most of what an agent costs. This piece walks through the formula behind my agent cost calculator, with a worked example computed by the same code the calculator runs.

Every turn re-sends everything

Split a request into three parts:

  • The stable prefix. The system prompt, tool definitions and any files the agent always carries. It is the same on every turn.
  • The history. Every earlier turn: what went in, and what the model wrote back.
  • The new input. This turn’s user message or tool result.

So the request at turn t is:

request(t) = prefix + (t − 1) × (newIn + out) + newIn

newIn is the new input per turn and out is the output per turn. The history term grows by one turn’s worth of input and output every step. Without caching, every one of those tokens is billed at the full input price, on every turn.

Here is how the request grows with the calculator’s default shape: a 50,000-token prefix, 500 new input tokens and 800 output tokens per turn.

TurnTokens sent in that request
150,500
251,800
1062,200
2075,200

The model only wrote 16,000 tokens across the whole conversation. It read 1,257,000.

What caching changes

Prompt caching lets the provider keep the processed form of a prompt’s beginning for a while. If the next request starts with exactly the same tokens, that part is a cache read: billed at a small fraction of the input price. Tokens that are new since the last request are a cache write: stored for next time and, on some providers, billed at a bit more than the input price.

In an agent loop that fits neatly. Everything in request t was already in request t − 1, except the last turn’s output and this turn’s new input. So:

  • Turn 1 writes the whole first request: the prefix plus the first input. Where writes carry a premium, it costs slightly more than without caching.
  • Every later turn reads everything the previous request sent, and writes only what was added since.

That is all the calculator models. In code, per turn:

const read = t === 1 ? 0 : prevRequest;  // already cached
const write = request - read;             // new since the last request

A worked example

I’ll use the calculator’s defaults: 20 turns, 1,000 conversations a month, and prices per million tokens of $2 input, $10 output, $0.1 cache read and $2.5 cache write.

One conversationWithout cachingWith caching
Input tokens at full price1,257,000none
Cache readsnone1,181,800
Cache writesnone75,200
Output tokens16,00016,000
Cost per conversation$2.67$0.466
Per month (1,000 conversations)$2,674$466

Two things stand out. Without caching, output is only 6% of the bill, even at five times the input price per token, because the model writes little and reads a lot. And caching wins by turning repeated reading into cheap reading: 1,181,800 of the 1,257,000 input tokens are reads. Once input is that cheap, output becomes 34% of the smaller bill.

The cumulative cost per turn shows the shape:

$0$1$2$315101520without caching $2.67with caching $0.466turn
Cumulative cost of one 20-turn conversation, computed with the calculator's code. Turn 1 costs $0.134 with caching against $0.109 without: the write premium. Every turn after that is cheaper. © Kernel & Chai

Without caching, the line bends upward: each turn costs more than the last, because the history keeps growing. With caching, it is nearly straight. Each turn pays cheap reads for the history plus a small write for what’s new.

The warm-cache assumption

The calculator assumes the cache is still there on the next turn. That is the best case, and it is where real bills drift away from the estimate.

Caches expire. Providers document different lifetimes and controls, and they change, so read the current page for the model you use. Here is what three providers’ caching docs said when I wrote this:

What the docs sayClaude APIOpenAI APIGemini API
How it startsa cache_control field: one top-level field for automatic placement, or one per blockon by default for supported modelsimplicit caching on by default for Gemini 2.5 and newer; explicit caches for some models
Lifetime5 minutes by default, refreshed on each use; 1 hour at extra cost30 minutes after the last write or reuse, for GPT-5.6 and laternot stated on the caching page
Cache write1.25× the input price (2× for the 1-hour cache)1.25× the input price, for GPT-5.6 and laternot stated on the caching page
Cache read0.1× the input price; some newer models are lower0.1× for GPT-5.6 and later; “up to 95%” off cached inputsavings passed on automatically when a request hits the cache
Minimum prompt512 to 4,096 tokens, depending on the model1,024 tokens, for GPT-5.6 and later2,048 to 4,096 tokens, depending on the model

A turn that comes after the cache has expired pays a write for the whole request again. An agent that waits minutes for a slow tool, or a chat where the user wanders off for lunch, will pay more than this model says.

How to keep the cache working

The rules follow from how a cache match works. It only reuses an identical prefix.

  • Put stable content first. Instructions, tool definitions and reference files go at the top. Anything that changes per request, such as a timestamp or the user’s name, goes at the end. OpenAI’s guide says the same.
  • Don’t edit history. Rewriting or reordering earlier turns breaks the prefix, and the next request pays writes again.
  • Know your minimum. Prompts shorter than the provider’s minimum aren’t cached at all. The Claude docs note that no error is returned, so it fails quietly.
  • Measure it. Compare a week of real usage with the model above before you trust the monthly number.

To try your own numbers (a coding agent with long tool outputs looks very different from a support chat), open the agent cost calculator. It runs the same conversationCost function as this page.

Sources

the newsletter

One explainer a week.

The week's video, written out properly, with sources. Nothing else. Unsubscribe in one click.

Watch on YouTube ↗