HumanEval is a code-generation benchmark introduced by OpenAI in 2021, consisting of 164 hand-written Python programming problems. Each problem includes a function signature, a docstring describing the required behavior, and a set of unit tests. A model is scored on pass@k — the probability that at least one of k generated solutions passes all unit tests — with pass@1 (a single attempt must be correct) being the most commonly cited variant. Because the problems range from simple string manipulation to algorithmic tasks involving recursion and data structures, HumanEval tests both syntactic correctness and logical reasoning about program behavior.

Think of it like a coding interview with 164 questions: the model gets the problem statement and must write working code, not just describe what the code should do. The unit tests act as an objective, automated grader that cannot be fooled by plausible-looking but broken implementations.

A typical HumanEval problem asks a model to implement a function like def has_close_elements(numbers, threshold) that checks whether any two elements in a list are closer than a given threshold. The model must return syntactically valid Python that passes the hidden test suite. GPT-4 achieves roughly 85% pass@1; smaller open-source models often score 40-65%, depending on training data and model size.

HumanEval benchmarks the same Inference-time capabilities that matter for Prompt Engineering in coding assistants; it complements benchmarks like MMLU (which tests general knowledge) by focusing specifically on executable output rather than multiple-choice answers. Results on HumanEval are frequently cited alongside the model's Context Window and Parameter count in evaluation reports.

You will encounter HumanEval scores when comparing coding-focused LLMs, evaluating whether a model is suitable for autocomplete or agent-based coding tasks, or reading research papers that introduce new code-generation techniques.