Top-k sampling is a decoding strategy for language models in which, at each generation step, the probability distribution over the entire vocabulary is truncated to only the k highest-scoring tokens before a token is sampled. Tokens outside the top-k are assigned zero probability, preventing the model from ever selecting very unlikely words. The value of k is a tunable Hyperparameter: low values (k=1 becomes greedy decoding) produce deterministic, conservative output, while higher values (k=50–100) allow more creative variation.

Think of it like a quiz show where only the top five finalists are allowed on stage—the audience (the model) still picks randomly among those five, but the long tail of hopeless contestants has already been eliminated. You get diversity without total chaos.

from transformers import pipeline
generator = pipeline("text-generation", model="gpt2")
output = generator(
    "The best way to learn programming is",
    do_sample=True,
    top_k=50,
    max_new_tokens=60
)
print(output[0]["generated_text"])

Setting top_k=50 keeps the fifty most likely next tokens in play, balancing fluency with variety.

Top-k sampling is closely related to Top-p (Nucleus Sampling), which uses a cumulative probability threshold instead of a fixed count, and to Temperature, which reshapes the distribution before either truncation strategy is applied. The two truncation methods are often combined.

You'll encounter top-k sampling in text-generation APIs, image-caption models, and any scenario where you need to tune the creativity-versus-coherence trade-off of an LLM without retraining it.