GLUE
In plain wordsA 2018 bundle of nine English language-understanding tests combined into one score, which became the yardstick for the first wave of pretrained language models such as BERT.
A natural language understandingnatural language understanding, n.: The part of language AI concerned with working out what a piece of text means — who did what to whom, what the writer wants and what follows from it — rather than with producing text. benchmark that scores one system on nine English classification and regression tasks reused from earlier datasets — grammatical acceptability (CoLA), sentiment (SST-2), paraphrase and semantic similarity (MRPC, STS-B, QQP) and natural-language inference (MNLI, QNLI, RTE, WNLI) — and posts the average to a public leaderboard. Any architecture may enter, but several tasks have little training data, so systems that transfer what they learn between tasks do best; a separate expert-built diagnostic set probes specific linguistic phenomena. GLUE became the yardstick for pretrained transfer learningtransfer learning, n.: Reusing what a model learned on one task to get a head start on another — like a pianist picking up the organ faster than a beginner. It lets models learn new jobs from little data. models: BERTBERT, prop. n.: An influential 2018 Google language model that learned to understand text by filling in hidden words. It made search engines and many language tools much better at grasping meaning in context. (October 2018) raised the state of the art by several points, and by mid-2019 systems exceeded the estimated non-expert human baseline of 87.1, prompting the harder SuperGLUESuperGLUE, prop. n.: A harder follow-up to the GLUE language-understanding test, released in 2019 after AI systems had already beaten the human baseline on the original.. Its rapid exhaustion is a textbook case of benchmark saturationbenchmark saturation, n.: When the best AI systems score so close to the maximum on a test that it can no longer tell them apart or show further progress..
Fine-tuning BERT-base on each GLUE task takes under an hour on a single GPU.
Origin
Introduced by Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy and Samuel R. Bowman (New York University, the University of Washington and DeepMind) in April 2018, and published at ICLR 2019.
Usage note
The small WNLI task was adversarial for that era's models, which often scored below the majority-class baseline; the BERT paper excluded it from its reported results.
Related
- SuperGLUESuperGLUE, prop. n.: A harder follow-up to the GLUE language-understanding test, released in 2019 after AI systems had already beaten the human baseline on the original.
- benchmark saturationbenchmark saturation, n.: When the best AI systems score so close to the maximum on a test that it can no longer tell them apart or show further progress.
- BERTBERT, prop. n.: An influential 2018 Google language model that learned to understand text by filling in hidden words. It made search engines and many language tools much better at grasping meaning in context.
- natural language understandingnatural language understanding, n.: The part of language AI concerned with working out what a piece of text means — who did what to whom, what the writer wants and what follows from it — rather than with producing text.
- transfer learningtransfer learning, n.: Reusing what a model learned on one task to get a head start on another — like a pianist picking up the organ faster than a beginner. It lets models learn new jobs from little data.
- leaderboardleaderboard, n.: A public ranking table showing which AI systems score best on a test, updated as new results arrive — like a sports league table for models.
Sources
- Wang, A., Singh, A., Michael, J., Hill, F., Levy, O. & Bowman, S. R. (2019). GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding. ICLR 2019. arxiv.org/abs/1804.07461
Mentioned in
benchmarkbenchmark, n.: A fixed, shared test — a set of questions or tasks with an agreed way of scoring — used to compare AI systems with one another and to track progress over time., benchmark saturationbenchmark saturation, n.: When the best AI systems score so close to the maximum on a test that it can no longer tell them apart or show further progress., inferenceinference, n.: Drawing a conclusion from evidence. In today's AI it most often means running an already-trained model to get an answer, as opposed to training it., natural language understandingnatural language understanding, n.: The part of language AI concerned with working out what a piece of text means — who did what to whom, what the writer wants and what follows from it — rather than with producing text., natural language inferencenatural language inference, n.: Deciding whether one sentence follows from another, contradicts it, or neither: if 'a man is playing a guitar on stage', is it true that 'a man is performing music'?, semantic textual similaritysemantic textual similarity, n.: How close two sentences are in meaning, scored from 'about different things' to 'say exactly the same thing' — and the tests that check whether a computer's scores match people's judgments., SuperGLUESuperGLUE, prop. n.: A harder follow-up to the GLUE language-understanding test, released in 2019 after AI systems had already beaten the human baseline on the original.