BERTScore

A metric for evaluating generated text — translations, captions, summaries — against a reference by comparing contextual embeddingsembedding, n.: A list of numbers that stands for a word, picture or other item, arranged so that similar items get similar lists. It lets a computer measure similarity of meaning as distance. rather than surface n-grams. Both texts are encoded with a pretrained transformer such as BERTBERT, prop. n.: An influential 2018 Google language model that learned to understand text by filling in hidden words. It made search engines and many language tools much better at grasping meaning in context. or RoBERTa; each candidate token is matched to its most similar reference token by cosine similaritycosine similarity, n.: A score from −1 to 1 for how closely two lists of numbers point the same way, ignoring their size. It is the standard test of whether two texts mean similar things. (greedy matching), and the average best-match similarity gives precision, the reverse direction gives recall, and their harmonic mean gives an F1 score. Optional inverse-document-frequency weighting emphasizes rare, informative words, and rescaling against a baseline makes scores easier to read. Because embeddings capture synonyms and paraphrase, BERTScore correlates better with human judgments than BLEUBLEU, abbr.: An automatic score for machine translation that checks how many short runs of words in the computer's translation also appear in human reference translations. or ROUGEROUGE, abbr.: An automatic score for summaries that checks how much of the wording in human-written reference summaries also appears in the computer-generated one. on many tasks, though it can still reward fluent text that contradicts the reference, and its values depend on the encoder model and layer chosen.

The paraphrasing model scores poorly on BLEU but close to the reference on BERTScore.