SuperGLUE
In plain wordsA harder follow-up to the GLUE language-understanding test, released in 2019 after AI systems had already beaten the human baseline on the original.
A benchmark and leaderboard for English natural language understandingnatural language understanding, n.: The part of language AI concerned with working out what a piece of text means — who did what to whom, what the writer wants and what follows from it — rather than with producing text. launched as a 'stickier' replacement for GLUEGLUE, prop. n.: A 2018 bundle of nine English language-understanding tests combined into one score, which became the yardstick for the first wave of pretrained language models such as BERT., whose non-expert human baseline systems had already overtaken. It retains the single-score, multi-task design but swaps in eight more demanding tasks emphasizing reasoning, commonsensecommonsense reasoning, n.: The everyday background knowledge and reasoning people take for granted — dropped glasses break, hungry people eat, nobody is in two places at once — which machines long struggled to match. and coreference: BoolQ (yes/no questions), CB (CommitmentBank inference), COPA (causal reasoning), MultiRC (multi-sentence reading comprehension), ReCoRD (cloze-style reading requiring commonsense), RTE (textual entailment), WiC (word sense in context) and WSC (the Winograd schema challengeWinograd schema challenge, prop. n.: A test of machine common sense built from pairs of sentences in which changing one word changes what a pronoun refers to — something people resolve instantly using everyday knowledge.), plus diagnostic sets for linguistic phenomena and gender bias. Its human baseline was 89.8; in January 2021 Microsoft's DeBERTa and a Google system combining T5 and Meena both exceeded it. SuperGLUE's saturation closed the era of single-score NLU leaderboards, as attention shifted to benchmarks for large language models such as MMLUMMLU, prop. n.: A giant multiple-choice exam for AI covering 57 subjects, from school math and history to law and medicine; for years it was the most quoted test of a model's general knowledge. (2020).
SuperGLUE scores above the human baseline did not translate into robust reading comprehension in deployment.
Origin
Introduced by Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy and Samuel R. Bowman in May 2019, and published at NeurIPS 2019.
Usage note
Exceeding the human baseline here means beating crowdworkers' average under the benchmark's conditions, not human-level language understanding — a caution the creators themselves voiced.
Related
- GLUEGLUE, prop. n.: A 2018 bundle of nine English language-understanding tests combined into one score, which became the yardstick for the first wave of pretrained language models such as BERT.
- benchmark saturationbenchmark saturation, n.: When the best AI systems score so close to the maximum on a test that it can no longer tell them apart or show further progress.
- Winograd schema challengeWinograd schema challenge, prop. n.: A test of machine common sense built from pairs of sentences in which changing one word changes what a pronoun refers to — something people resolve instantly using everyday knowledge.
- natural language understandingnatural language understanding, n.: The part of language AI concerned with working out what a piece of text means — who did what to whom, what the writer wants and what follows from it — rather than with producing text.
- BERTBERT, prop. n.: An influential 2018 Google language model that learned to understand text by filling in hidden words. It made search engines and many language tools much better at grasping meaning in context.
- T5T5, prop. n.: A 2019 Google model that treats every language task — translating, summarizing, answering, classifying — as turning one piece of text into another, using one shared design.
Sources
- Wang, A., Pruksachatkun, Y., Nangia, N. et al. (2019). SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems. NeurIPS 2019. arxiv.org/abs/1905.00537
Mentioned in
benchmark saturationbenchmark saturation, n.: When the best AI systems score so close to the maximum on a test that it can no longer tell them apart or show further progress., GLUEGLUE, prop. n.: A 2018 bundle of nine English language-understanding tests combined into one score, which became the yardstick for the first wave of pretrained language models such as BERT., natural language understandingnatural language understanding, n.: The part of language AI concerned with working out what a piece of text means — who did what to whom, what the writer wants and what follows from it — rather than with producing text.