How Claude's prompt caching works — and how not to overpay

Prompt caching can cut the cost of repeated calls for real — but only if the cached prefix is byte-for-byte identical between requests. This page explains where to put the cache breakpoint, when to prefer RAG over long context, and how to handle the errors the API returns.

Where to put the cache breakpoint

An API call is rendered in this order: tools, then system, then messages. A cache breakpoint on the last block of the system prompt caches the entire prefix that came before it — tools and system together — as long as that prefix is byte-for-byte identical between requests.

The most common mistake is placing the breakpoint after content that varies per request: a unique user message never repeats exactly, so every call writes a fresh cache entry, paying a write premium, and never gets a hit. Variable data always goes last, after anything that repeats between calls.

One byte changes and the whole cache invalidates

Caching isn't approximate: any difference before the breakpoint — a timestamp in the system prompt, a different order of tools — invalidates everything that comes after it. There's no partially valid cache; either the prefix is identical and gets reused, or it isn't and gets reprocessed from scratch.

This also decides the strategy when calls aren't continuous: if the gap between batches of calls exceeds the cache's default TTL but still stays under an hour, there's a longer-TTL option built exactly for that case — the alternative to simply accepting that the cache expires between batches.

429, 529, and 400 call for different reactions

429 (rate limit) and 529 (overloaded) are transient errors: the right response is to retry with exponential backoff, because the same request has a good chance of succeeding on the next attempt. A 400, by contrast, is a bug in the request — retrying without fixing anything just reproduces the same error indefinitely.

A pipeline that already handles all three should also read the rate-limit response headers on every successful call, to do proactive throttling before hitting a 429 — instead of finding out about the limit only after already blowing past it.

When long context isn't the right answer

Long context works well until the corpus grows beyond any available context window — when that happens, RAG scales in a way that simply enlarging the context doesn't. The trade-off has a cost: RAG fragments the corpus into retrievable chunks, which works well for pinpoint search but gets in the way of exactly the questions that require synthesizing across several documents at once.

The choice between the two isn't about which is "better" — it's about the size of the corpus and the type of question the use case actually asks.

Compaction: why the client still has to manage state

When a long conversation exceeds the context window, Claude Code can compact the history, summarizing older exchanges to free up space. This preserves the conversation's continuity, but it isn't automatic on the client side: on every turn, it's still the responsibility of whoever builds on the API to correctly resend the session identifier and the returned history — or the state gets lost even with compaction enabled.

It's the same principle as RAG versus long context, seen from another angle: managing what fits in the context window is an ongoing responsibility, not a setting you flip once and forget.