exact match
In plain wordsThe simplest way to grade an AI's answer: it counts as right only if it is identical to the reference answer, after minor tidying such as ignoring capital letters.
A scoring rule under which an answer earns credit only if, after light normalization, it is identical character for character to a reference string — or to any one of several acceptable references — with the score reported as the share of items that earn credit. Normalization typically ignores case and strips articles such as 'the', punctuation and redundant spaces. Exact match suits short, closed-form answers such as numbers, names or multiple-choice letters, and is used in extractive question answeringquestion answering, n.: Systems that reply to people's questions with the right answer, whether by finding it in a passage they are given, searching a database or the web, or drawing on what they already know., math benchmarks such as GSM8KGSM8K, prop. n.: A set of about 8,500 grade-school math word problems, each needing a few steps of simple arithmetic, used to test whether language models can reason through a problem step by step., and the short-answer items of Humanity's Last ExamHumanity's Last Exam, prop. n.: A test of 2,500 extremely hard questions with definite answers, written by about a thousand experts worldwide and billed as the last academic exam AI would need; at launch, top models scored under 10%.. It is strict and brittle: a correct answer phrased differently ('4' versus 'four', or a full sentence) scores zero. It is therefore often paired with token-overlap F1F1 score, n.: A single score that combines how many of a model's alarms are right and how many real cases it catches, and stays low unless both are good., rules that extract the final answer, or equivalence checks by a computer algebra system or an LLM-as-a-judgeLLM-as-a-judge, n.: Using one AI language model to grade or compare the answers of another, as a cheaper and faster stand-in for human reviewers..
Exact match dropped eleven points when the model started answering in full sentences.
Origin
String-match scoring is as old as automatic evaluation; Pranav Rajpurkar and colleagues' SQuADSQuAD, prop. n.: A Stanford collection of more than 100,000 questions about Wikipedia articles, each solved by highlighting the words in the article that give the answer; it drove rapid progress in machine reading from 2016. (2016) popularized 'Exact Match (EM)', reported alongside F1, as a named headline metric for machine reading comprehension.
Usage note
A low exact-match score can reflect formatting rather than knowledge; reports should state the normalization and answer-extraction rules used.
Related
- SQuADSQuAD, prop. n.: A Stanford collection of more than 100,000 questions about Wikipedia articles, each solved by highlighting the words in the article that give the answer; it drove rapid progress in machine reading from 2016.
- F1 scoreF1 score, n.: A single score that combines how many of a model's alarms are right and how many real cases it catches, and stays low unless both are good.
- accuracyaccuracy, n.: The share of a model's answers that are right, such as 95 out of 100. Simple and intuitive, but misleading when one kind of answer is far more common than the others.
- question answeringquestion answering, n.: Systems that reply to people's questions with the right answer, whether by finding it in a passage they are given, searching a database or the web, or drawing on what they already know.
- pass@kpass@k, n.: A score for code-writing AI: the chance that at least one of k attempts at a problem is correct, as checked by actually running tests on the code.
- LLM-as-a-judgeLLM-as-a-judge, n.: Using one AI language model to grade or compare the answers of another, as a cheaper and faster stand-in for human reviewers.
Sources
- Rajpurkar, P., Zhang, J., Lopyrev, K. & Liang, P. (2016). SQuAD: 100,000+ Questions for Machine Comprehension of Text. EMNLP 2016. arxiv.org/abs/1606.05250
Mentioned in
accuracyaccuracy, n.: The share of a model's answers that are right, such as 95 out of 100. Simple and intuitive, but misleading when one kind of answer is far more common than the others., benchmarkbenchmark, n.: A fixed, shared test — a set of questions or tasks with an agreed way of scoring — used to compare AI systems with one another and to track progress over time., Humanity's Last ExamHumanity's Last Exam, prop. n.: A test of 2,500 extremely hard questions with definite answers, written by about a thousand experts worldwide and billed as the last academic exam AI would need; at launch, top models scored under 10%., AIMEAIME, prop. n.: A hard US high-school math contest whose 15 questions each have a whole-number answer; AI labs use each year's new papers to test their models' mathematical reasoning., BLEUBLEU, abbr.: An automatic score for machine translation that checks how many short runs of words in the computer's translation also appear in human reference translations., GAIAGAIA, prop. n.: A test of AI assistants made of real-world questions that people can answer with a browser and some patience, but that require an AI to search, read files and combine facts., GSM8KGSM8K, prop. n.: A set of about 8,500 grade-school math word problems, each needing a few steps of simple arithmetic, used to test whether language models can reason through a problem step by step., LLM-as-a-judgeLLM-as-a-judge, n.: Using one AI language model to grade or compare the answers of another, as a cheaper and faster stand-in for human reviewers., MATHMATH, prop. n.: A set of 12,500 problems from high-school math competitions, each with a full worked solution, used to test language models on genuinely challenging mathematics., pass@kpass@k, n.: A score for code-writing AI: the chance that at least one of k attempts at a problem is correct, as checked by actually running tests on the code., question answeringquestion answering, n.: Systems that reply to people's questions with the right answer, whether by finding it in a passage they are given, searching a database or the web, or drawing on what they already know., SQuADSQuAD, prop. n.: A Stanford collection of more than 100,000 questions about Wikipedia articles, each solved by highlighting the words in the article that give the answer; it drove rapid progress in machine reading from 2016.