Prompting
Top-k Sampling
A decoding strategy that picks the next token only from the top K most likely candidates — trading diversity for focus.
01 ——
In plain English
Top-k sampling is one of the basic levers for controlling LLM output randomness. At each step, the model produces a probability for every possible next token. Top-k throws away all but the K highest-probability tokens and samples from those.
How it works:
- K=1 is equivalent to greedy decoding (always pick the most likely token)
- K=40 is a common default — enough diversity, still focused
- K=∞ means no restriction; you're back to sampling from the full distribution
- Lower K = more deterministic, higher K = more diverse
Where you see it:
Most chat APIs (OpenAI, Anthropic, Google, OpenRouter) expose top_k as a parameter, though defaults are usually fine. Worth tuning if you're getting repetitive output (raise K) or too-creative drift (lower K).
Top-k vs top-p:
- Top-k is a fixed count (always K candidates)
- Top-p is a fixed cumulative probability (varies by how peaked the distribution is)
In practice, top-p is more commonly tuned in production because it adapts to the model's confidence.
02 ——
Related terms
Top-p Sampling
A decoding strategy (also called nucleus sampling) that picks the next token from the smallest set of candidates whose cumulative probability exceeds P.
Temperature
A setting that controls how random or creative an AI model's responses are — lower values produce focused answers, higher values produce more varied ones.
Prompt Engineering
The practice of crafting inputs to an AI model carefully to get better, more reliable outputs.
Inference
The process of running a trained AI model to generate a response — as opposed to training the model.
Tokens
The basic units of text that AI models read and write — roughly ¾ of a word each. Models are priced and limited by token count.
Streaming
Sending an AI model's response token-by-token as it's generated, so the user sees text appear immediately instead of waiting for the full reply.
Back to glossaryLast reviewed May 2026