Bilingual Evaluation Understudy (BLEU) is an automatic metric for evaluating the quality of machine-generated text—most commonly machine translation—by measuring how closely it matches one or more human-written reference texts. It computes a precision score over overlapping n-grams (sequences of one to four consecutive words) between the candidate output and references, then applies a brevity penalty to discourage artificially short outputs. Scores range from 0 to 1, where 1 indicates a perfect match with the reference.

Think of it like grading a student's essay by counting how many of their phrases appear verbatim in a model answer key. The more matching phrases, the higher the grade—though the student still loses marks if their essay is suspiciously short.

In practice, a translation system producing "The cat sat on the mat" would score well against a reference of "The cat is sitting on the mat" because several bigrams overlap. BLEU is computed per corpus, so individual sentence scores are unreliable; researchers typically report corpus-level scores. A BLEU of 0.35 on a standard benchmark like WMT is considered competitive for neural machine translation.

from nltk.translate.bleu_score import corpus_bleu
references = [[["the", "cat", "sat", "on", "the", "mat"]]]
hypotheses = [["the", "cat", "is", "on", "the", "mat"]]
print(corpus_bleu(references, hypotheses))  # e.g. 0.57

BLEU is closely related to Perplexity as both are automatic evaluation proxies; however, neither captures fluency or meaning as well as human judgment. It is often paired with ROUGE (for summarization), and its limitations have motivated newer metrics such as BERTScore, which uses Embedding similarity to assess semantic overlap rather than surface n-gram matches. Understanding BLEU is prerequisite knowledge for evaluating any LLM or sequence-to-sequence model.

You will encounter BLEU scores in machine translation papers, summarization benchmarks, and leaderboards comparing Foundation Models on generation tasks.