natural language inference
In plain wordsDeciding whether one sentence follows from another, contradicts it, or neither: if 'a man is playing a guitar on stage', is it true that 'a man is performing music'?
The task of determining the logical relationship between a premise and a hypothesis: entailment (the hypothesis must be true if the premise is), contradiction (it must be false) or neutral (neither follows). It serves as a test of natural language understandingnatural language understanding, n.: The part of language AI concerned with working out what a piece of text means — who did what to whom, what the writer wants and what follows from it — rather than with producing text. requiring lexical knowledge, commonsense reasoningcommonsense reasoning, n.: The everyday background knowledge and reasoning people take for granted — dropped glasses break, hungry people eat, nobody is in two places at once — which machines long struggled to match. and the handling of negation, quantifiers and reference. Large crowd-sourced datasets enabled neural NLI models, which were later found to exploit annotation artifacts — for example, negation words signaling contradiction — prompting harder adversarial datasets. NLI models also serve as tools: for zero-shotzero-shot learning, n.: Doing a task without being shown a single example of it — say, an AI sorting reviews into positive and negative just from being told to, or recognizing an animal it never saw in training. classification (is 'this text is about sports' entailed?), for checking whether a summary or generated claim is supported by its source, and for clustering answers in semantic entropysemantic entropy, n.: A way to tell when an AI is likely making something up: ask it the same question several times and check whether the answers mean the same thing. Answers that keep changing signal guessing..
We use an NLI model to flag summary sentences that the source document doesn't entail.
Origin
Ido Dagan, Oren Glickman and Bernardo Magnini launched the PASCAL Recognising Textual Entailment challenge in 2005. The name 'natural language inference', used in Bill MacCartney and Christopher Manning's work on natural logic from 2008, became standard with Samuel Bowman and colleagues' Stanford NLI corpus (SNLI, 2015; about 570,000 sentence pairs) and MultiNLI (2018).
Related
- natural language understandingnatural language understanding, n.: The part of language AI concerned with working out what a piece of text means — who did what to whom, what the writer wants and what follows from it — rather than with producing text.
- GLUEGLUE, prop. n.: A 2018 bundle of nine English language-understanding tests combined into one score, which became the yardstick for the first wave of pretrained language models such as BERT.
- commonsense reasoningcommonsense reasoning, n.: The everyday background knowledge and reasoning people take for granted — dropped glasses break, hungry people eat, nobody is in two places at once — which machines long struggled to match.
- zero-shot learningzero-shot learning, n.: Doing a task without being shown a single example of it — say, an AI sorting reviews into positive and negative just from being told to, or recognizing an animal it never saw in training.
- semantic entropysemantic entropy, n.: A way to tell when an AI is likely making something up: ask it the same question several times and check whether the answers mean the same thing. Answers that keep changing signal guessing.
- factualityfactuality, n.: How far what an AI says is actually true about the world. A factual model gets facts, numbers, names and dates right and does not make things up.
Sources
- Dagan, I., Glickman, O. & Magnini, B. (2006). The PASCAL Recognising Textual Entailment Challenge. In Machine Learning Challenges, LNCS 3944. Springer.
- Bowman, S. R., Angeli, G., Potts, C. & Manning, C. D. (2015). A large annotated corpus for learning natural language inference. EMNLP 2015.
Mentioned in
natural language understandingnatural language understanding, n.: The part of language AI concerned with working out what a piece of text means — who did what to whom, what the writer wants and what follows from it — rather than with producing text., semantic entropysemantic entropy, n.: A way to tell when an AI is likely making something up: ask it the same question several times and check whether the answers mean the same thing. Answers that keep changing signal guessing., semantic textual similaritysemantic textual similarity, n.: How close two sentences are in meaning, scored from 'about different things' to 'say exactly the same thing' — and the tests that check whether a computer's scores match people's judgments.