Infra & cost
Prompt Caching
A feature that stores parts of a prompt the model has already processed, making repeat or follow-up requests much faster and cheaper.
01 ——
In plain English
Prompt caching is an inference optimisation where the AI provider stores the model's intermediate computation for the long, repeated parts of your prompt (system instructions, large documents, conversation history). On subsequent requests with the same prefix, the model skips re-processing it.
Why it matters:
- Lower cost — cached tokens are typically 50–90% cheaper
- Faster responses — time to first token can drop dramatically
- Practical for RAG and agents — repeated context is the norm
When to use it:
- Long system prompts or instructions
- Document Q&A where many users ask different questions about the same doc
- Agent loops that re-read the same tool definitions or memory
- Multi-turn conversations
Supported by: Anthropic, OpenAI, Google, and most major providers, with slightly different APIs and pricing for cached vs fresh tokens.
02 ——
Related terms
Inference
The process of running a trained AI model to generate a response — as opposed to training the model.
Latency
The time it takes an AI model to respond to a request — from when you hit send to when the first or final word appears.
API
An interface that lets developers send requests to an AI model and get responses programmatically — the way most AI tools talk to LLMs.
Context Window
The maximum amount of text (tokens) an AI model can read and remember at once during a single conversation.
LLM
Large Language Model — the type of AI behind tools like ChatGPT and Claude, trained to understand and generate text.
Back to glossaryLast reviewed May 2026