Infra & cost
Inference
The process of running a trained AI model to generate a response — as opposed to training the model.
01 ——
In plain English
Inference is what happens when you actually use an AI model. You send it a prompt, the model processes it through billions of parameters, and it outputs a response. Every time you chat with an AI, that's inference.
Training vs inference:
- Training = teaching the model (done once, very expensive, requires massive compute)
- Inference = using the model (done billions of times per day, the ongoing cost)
Why it matters for AI tools: Inference cost is the main operating expense for AI products. Faster, cheaper inference (via model compression, batching, or specialised hardware) is what makes AI tools affordable at scale. When a provider mentions "tokens per second" or "latency", they're talking about inference performance.
02 ——
Related terms
LLM
Large Language Model — the type of AI behind tools like ChatGPT and Claude, trained to understand and generate text.
Tokens
The basic units of text that AI models read and write — roughly ¾ of a word each. Models are priced and limited by token count.
Latency
The time it takes an AI model to respond to a request — from when you hit send to when the first or final word appears.
Streaming
Sending an AI model's response token-by-token as it's generated, so the user sees text appear immediately instead of waiting for the full reply.
Quantization
Shrinking an AI model by storing its weights in lower-precision numbers — making it smaller, faster, and cheaper with minimal quality loss.
Back to glossaryLast reviewed May 2026