Skip to content
Last Dictionary
Plane
Evaluation & BenchmarksLanguage Models & Transformers

BLEU

abbreviation/BLOO/Also: BLEU score, Bilingual Evaluation Understudy, SacreBLEU

In plain wordsAn automatic score for machine translation that checks how many short runs of words in the computer's translation also appear in human reference translations.

  1. A corpus-level metric for machine-generated text, especially machine translationmachine translation, n.: Software that renders writing or speech in one language into another language, as online translation services and phone translation apps do., based on n-gram overlap with one or more human reference translations. For n from 1 to 4 it computes modified precision p_n — the fraction of candidate n-grams found in a reference, with each n-gram's count clipped at its maximum count in any single reference — and combines these by a weighted geometric mean. A brevity penalty BP lowers the score when the candidate corpus length c is shorter than the reference length r, since precision alone would reward terse output. Scores run from 0 to 1 and are usually reported ×100. BLEU is cheap, language-independent and tracks human judgments reasonably at the system level, but it ignores meaning-preserving paraphrase and is unreliable for single sentences; learned metrics such as COMET and BERTScoreBERTScore, prop. n.: An automatic score for computer-written text that compares meanings rather than exact words, using an AI language model to judge how closely each word matches the words of a reference text. agree better with human judgments.

    The new system gains 1.2 BLEU on the news test set, which is within run-to-run variance.

BLEU=BP⋅exp⁡(∑n=1Nwnlog⁡pn),BP=min⁡(1, e 1−r/c)\mathrm{BLEU} = \mathrm{BP}\cdot\exp\Big(\sum_{n=1}^{N} w_n \log p_n\Big),\quad \mathrm{BP} = \min\big(1,\, e^{\,1 - r/c}\big)

Origin

Introduced by Kishore Papineni, Salim Roukos, Todd Ward and Wei-Jing Zhu of IBM's Thomas J. Watson Research Center at ACL 2002, after circulating as a 2001 IBM research report. The name, Bilingual Evaluation Understudy, casts the metric as a stand-in for human judges. Matt Post's SacreBLEU (2018) standardized how it is computed and reported.

Usage note

BLEU depends on tokenization and on the reference set, so values from different papers are often not comparable; SacreBLEU was created to address this. It was not designed for open-ended generation by language modelslarge language model, n.: A computer program trained on enormous amounts of text to predict the words that come next. At vast scale, that simple skill lets it chat, answer questions, translate, summarize and write code..

Related

See also
  • ROUGEROUGE, abbr.: An automatic score for summaries that checks how much of the wording in human-written reference summaries also appears in the computer-generated one.
  • BERTScoreBERTScore, prop. n.: An automatic score for computer-written text that compares meanings rather than exact words, using an AI language model to judge how closely each word matches the words of a reference text.
  • machine translationmachine translation, n.: Software that renders writing or speech in one language into another language, as online translation services and phone translation apps do.
  • n-gram modeln-gram model, n.: An early type of language model that predicts the next word from just the few words before it, using counts of how often each word followed those words in a large collection of text.
  • human evaluationhuman evaluation, n.: Judging an AI's outputs by asking people to rate or compare them, because for many tasks — writing, conversation, translation — no automatic score captures quality well.
  • exact matchexact match, n.: The simplest way to grade an AI's answer: it counts as right only if it is identical to the reference answer, after minor tidying such as ignoring capital letters.
Contrast with
  • human evaluationhuman evaluation, n.: Judging an AI's outputs by asking people to rate or compare them, because for many tasks — writing, conversation, translation — no automatic score captures quality well.

Sources

  1. Papineni, K., Roukos, S., Ward, T. & Zhu, W.-J. (2002). Bleu: a Method for Automatic Evaluation of Machine Translation. Proceedings of ACL 2002, 311–318. aclanthology.org/P02-1040
  2. Post, M. (2018). A Call for Clarity in Reporting BLEU Scores. Proceedings of the Third Conference on Machine Translation (WMT 2018).

Mentioned in

Association for Computational LinguisticsAssociation for Computational Linguistics, prop. n.: The scientific society at the center of research on language technology, from translation software to chatbots. It runs the field's leading conferences and a free online archive of their papers., human evaluationhuman evaluation, n.: Judging an AI's outputs by asking people to rate or compare them, because for many tasks — writing, conversation, translation — no automatic score captures quality well., image captioningimage captioning, n.: Having a computer write a short description of a picture, such as 'a dog catching a frisbee in a park', to help blind people, search engines and other AI systems., machine translationmachine translation, n.: Software that renders writing or speech in one language into another language, as online translation services and phone translation apps do., natural language generationnatural language generation, n.: Getting computers to produce human language — from turning a spreadsheet into a weather report to writing an essay or a chatbot's reply., ROUGEROUGE, abbr.: An automatic score for summaries that checks how much of the wording in human-written reference summaries also appears in the computer-generated one., BERTScoreBERTScore, prop. n.: An automatic score for computer-written text that compares meanings rather than exact words, using an AI language model to judge how closely each word matches the words of a reference text.