Winograd schema challenge
In plain wordsA test of machine common sense built from pairs of sentences in which changing one word changes what a pronoun refers to — something people resolve instantly using everyday knowledge.
A test of commonsense reasoningcommonsense reasoning, n.: The everyday background knowledge and reasoning people take for granted — dropped glasses break, hungry people eat, nobody is in two places at once — which machines long struggled to match. put forward in 2011–2012 as a substitute for the Turing testTuring test, n.: Alan Turing's proposed test of machine intelligence: a judge holds typed conversations with a hidden person and a hidden computer, and the computer passes if the judge cannot consistently pick it out.. Its items are twin sentences, identical except for a word or two, in which the swap flips what an ambiguous pronoun refers to, so picking the right referent takes everyday knowledge of the world: in the classic example, city councilmen refused demonstrators a permit because they feared violence (they = the councilmen) or because they advocated it (they = the demonstrators). Schemas were written so that simple word statistics or a web search would not reveal the answer. A standard set of 273 problems (WSC273) resisted computer systems for years; in the only formal competition, in 2016, the best system answered about 58 percent of the preliminary round's pronoun problems correctly. From 2019, transformertransformer, n.: The design behind today's chatbots and much of modern AI. It handles text by letting every word look at every other word and decide which of them matter for understanding it. language modelslanguage model, n.: A system that has learned how likely different sequences of words are, so it can predict the next word in a text and, by doing that repeatedly, write text of its own. fine-tuned on related data exceeded 90 percent on WSC273, and a larger, adversarially filtered successor, WinoGrande (2019), was built to resist such shortcuts. In 2023 two of the challenge's originators and their colleagues declared it defeated.
Once language models aced the Winograd schema challenge, the argument shifted to whether they had learned common sense or dataset artifacts.
Origin
Proposed by Hector Levesque at a 2011 AAAI Spring Symposium and developed with Ernest Davis and Leora Morgenstern in 'The Winograd Schema Challenge' (KR 2012). Named after Terry Winograd, whose 1972 councilmen example inspired it.
Usage note
Its fall illustrates benchmark saturationbenchmark saturation, n.: When the best AI systems score so close to the maximum on a test that it can no longer tell them apart or show further progress. and the difficulty of designing tests that cannot be passed through statistical shortcuts.
Related
- commonsense reasoningcommonsense reasoning, n.: The everyday background knowledge and reasoning people take for granted — dropped glasses break, hungry people eat, nobody is in two places at once — which machines long struggled to match.
- Turing testTuring test, n.: Alan Turing's proposed test of machine intelligence: a judge holds typed conversations with a hidden person and a hidden computer, and the computer passes if the judge cannot consistently pick it out.
- SHRDLUSHRDLU, prop. n.: A program from around 1970 that let people move toy blocks in a simulated world by typing ordinary English commands and questions, and that could explain what it had done and why.
- benchmark saturationbenchmark saturation, n.: When the best AI systems score so close to the maximum on a test that it can no longer tell them apart or show further progress.
- benchmarkbenchmark, n.: A fixed, shared test — a set of questions or tasks with an agreed way of scoring — used to compare AI systems with one another and to track progress over time.
- Turing testTuring test, n.: Alan Turing's proposed test of machine intelligence: a judge holds typed conversations with a hidden person and a hidden computer, and the computer passes if the judge cannot consistently pick it out.
Sources
- Levesque, H. J., Davis, E. & Morgenstern, L. (2012). The Winograd Schema Challenge. Proceedings of the 13th International Conference on Principles of Knowledge Representation and Reasoning (KR 2012).
- Kocijan, V., Davis, E., Lukasiewicz, T., Marcus, G. & Morgenstern, L. (2023). The Defeat of the Winograd Schema Challenge. Artificial Intelligence.
Mentioned in
Turing testTuring test, n.: Alan Turing's proposed test of machine intelligence: a judge holds typed conversations with a hidden person and a hidden computer, and the computer passes if the judge cannot consistently pick it out., AI-completeAI-complete, adj.: Describes a problem believed to be so hard that solving it would require general, human-level intelligence, such as flawless translation or truly understanding a story., commonsense reasoningcommonsense reasoning, n.: The everyday background knowledge and reasoning people take for granted — dropped glasses break, hungry people eat, nobody is in two places at once — which machines long struggled to match., natural language understandingnatural language understanding, n.: The part of language AI concerned with working out what a piece of text means — who did what to whom, what the writer wants and what follows from it — rather than with producing text., coreference resolutioncoreference resolution, n.: Working out which words in a text are talking about the same someone or something — for instance, that 'she', 'the senator' and 'Ms. Okafor' all mean one woman., HellaSwagHellaSwag, prop. n.: A test in which an AI must pick the most sensible ending for a description of an everyday scene; the wrong endings were machine-written to fool computers, so people found it easy and early AI did not., Loebner PrizeLoebner Prize, prop. n.: A yearly contest, held from 1991 to 2019, in which chatbots tried to convince judges they were human in short text chats. Its grand prize was never won, and critics called it a publicity stunt., SHRDLUSHRDLU, prop. n.: A program from around 1970 that let people move toy blocks in a simulated world by typing ordinary English commands and questions, and that could explain what it had done and why., SuperGLUESuperGLUE, prop. n.: A harder follow-up to the GLUE language-understanding test, released in 2019 after AI systems had already beaten the human baseline on the original., Terry WinogradTerry Winograd, prop. n.: An American computer scientist who built SHRDLU, a celebrated early program that conversed in English about a world of blocks, and later became a critic of symbolic AI.