What is Prompt Caching?
Prompt Caching: A model API feature that stores the processed start of a prompt so repeat requests sharing it cost less and respond faster.
Prompt caching lets a model provider reuse work it already did on the beginning of a prompt. When many requests share the same prefix (a long system prompt, tool definitions, a reference document, or earlier turns of a conversation), the provider caches the processed prefix. Later requests that start with the exact same content read from the cache, which is billed at a fraction of the normal input price and reduces time to first token.
Providers implement it differently: some cache automatically, others require you to mark cache breakpoints, and caches expire after a few minutes of inactivity unless refreshed or extended. To benefit, structure prompts with the stable content first and the changing content (the user's question, the specific document) last, because any change early in the prompt invalidates the cache for everything after it.
Why it matters when you build or buy AI
For agents and assistants with long instructions or many tools, caching is one of the simplest ways to cut cost per task and latency without touching quality. It takes prompt structure discipline: a timestamp or user name near the top of the system prompt can silently break caching on every request. Check cache hit rates in your usage data.
Example
A support agent has a 12,000-token system prompt with policies and tool definitions. With caching, every conversation after the first reuses it at a discounted rate, and only the customer's messages are billed at full price.
Related terms
More in Engineering
Book 20 minutes, or send a brief
A former CTO replies. You leave with a written plan and a fixed estimate.
Prefer LinkedIn? Connect with Harish.