A benchmark is a standardised evaluation task — or a collection of such tasks — used to measure and compare the capabilities of AI models under controlled, reproducible conditions. A good benchmark covers a well-defined skill (mathematical reasoning, reading comprehension, code generation), has a scoring rubric that does not require human judgement for every answer, and was held out from model training data so that high scores reflect genuine capability rather than memorisation. Widely cited LLM benchmarks include MMLU (knowledge across 57 academic subjects), HumanEval (functional correctness of Python code), GSM8K (multi-step arithmetic word problems), and GPQA (graduate-level science questions).

Think of it like a standardised exam in school. Every student sits the same paper under the same conditions, so scores are comparable across schools and years — even though the exam cannot capture every aspect of a student ability.

Running a model against a benchmark typically involves formatting each question into a prompt, collecting the model output, and scoring it automatically:

correct = 0
for item in benchmark_dataset:
    prompt = format_prompt(item["question"], item["choices"])
    answer = model.generate(prompt, temperature=0)  # greedy for reproducibility
    if normalize(answer) == normalize(item["answer"]):
        correct += 1
accuracy = correct / len(benchmark_dataset)

Benchmarks connect to Perplexity (an intrinsic metric that often moves independently of task benchmark scores), Zero-Shot and Chain of Thought evaluation (many benchmarks are run in both zero-shot and few-shot configurations), and Overfitting (contamination of benchmark data into pretraining is a persistent concern that inflates apparent scores).

You will encounter benchmarks in model release announcements, research papers comparing architectures, and procurement decisions where teams need objective evidence to choose between competing Foundation Models.