question answering
In plain wordsSystems that reply to people's questions with the right answer, whether by finding it in a passage they are given, searching a database or the web, or drawing on what they already know.
The task of returning correct answers to questions expressed in natural language. Settings are distinguished by where the answer comes from: extractive QA, or reading comprehension, selects a span from a supplied passage, as in SQuADSQuAD, prop. n.: A Stanford collection of more than 100,000 questions about Wikipedia articles, each solved by highlighting the words in the article that give the answer; it drove rapid progress in machine reading from 2016.; open-domain QA must first retrieve relevant documents from a large collection such as the web; knowledge-base QA translates the question into a structured query (semantic parsingsemantic parsing, n.: Translating a sentence into a precise, machine-executable form — such as a database query or a logical formula — so that a computer can act on what was asked.); and closed-book QA answers from parametric knowledgeparametric knowledge, n.: What an AI model 'knows' from training, stored in the numbers of its network rather than looked up in documents — like facts you have memorized rather than ones you check in a book. alone. Answers may be short facts, yes or no, multiple-choice letters or long-form explanations, and evaluation uses exact matchexact match, n.: The simplest way to grade an AI's answer: it counts as right only if it is identical to the reference answer, after minor tidying such as ignoring capital letters., token-overlap F1 or human and model judgment. Large language models with retrieval-augmented generationretrieval-augmented generation, n.: Giving an AI an open-book exam: before answering, it looks up relevant passages in a set of documents and bases its reply on them, often citing where each point came from. and deep researchdeep research, n.: An AI feature that spends minutes rather than seconds on a question — planning, running dozens of web searches, reading many sources — and then writes a long report with citations. agents now handle most QA, but multi-hop, unanswerable and long-tail questions remain hard.
Open-domain question answering improved once we added a reranker before the reader model.
Origin
BASEBALL (Bert Green Jr. and colleagues, 1961) answered questions about a season's games from a database, and William Woods's LUNAR, demonstrated in 1971, answered geologists' questions about Apollo moon rocks. The TREC question-answering track (from 1999), IBM WatsonIBM Watson, prop. n.: An IBM question-answering computer that beat two champions of the US quiz show Jeopardy! in 2011. IBM then sold business AI products under the Watson name, many of which disappointed.'s Jeopardy! win (2011) and SQuAD (2016) marked later eras.
Related
- SQuADSQuAD, prop. n.: A Stanford collection of more than 100,000 questions about Wikipedia articles, each solved by highlighting the words in the article that give the answer; it drove rapid progress in machine reading from 2016.
- retrieval-augmented generationretrieval-augmented generation, n.: Giving an AI an open-book exam: before answering, it looks up relevant passages in a set of documents and bases its reply on them, often citing where each point came from.
- information retrievalinformation retrieval, n.: The study and engineering of search: how to organize huge piles of documents so that the few that matter to someone's question can be found and ranked quickly. AI systems rely on it to look things up.
- parametric knowledgeparametric knowledge, n.: What an AI model 'knows' from training, stored in the numbers of its network rather than looked up in documents — like facts you have memorized rather than ones you check in a book.
- IBM WatsonIBM Watson, prop. n.: An IBM question-answering computer that beat two champions of the US quiz show Jeopardy! in 2011. IBM then sold business AI products under the Watson name, many of which disappointed.
- visual question answeringvisual question answering, n.: Asking an AI a question about a picture, such as 'How many people are wearing hats?', and having it answer in words.
- deep researchdeep research, n.: An AI feature that spends minutes rather than seconds on a question — planning, running dozens of web searches, reading many sources — and then writes a long report with citations.
Sources
- Green, B. F., Jr., Wolf, A. K., Chomsky, C. & Laughery, K. (1961). Baseball: An Automatic Question-Answerer. Proceedings of the Western Joint IRE-AIEE-ACM Computer Conference.
- Rajpurkar, P., Zhang, J., Lopyrev, K. & Liang, P. (2016). SQuAD: 100,000+ Questions for Machine Comprehension of Text. EMNLP 2016.
Mentioned in
natural language processingnatural language processing, n.: The field that teaches computers to handle human language — to read, interpret, translate, summarize and produce text and speech. It is one of the oldest and largest areas of artificial intelligence., exact matchexact match, n.: The simplest way to grade an AI's answer: it counts as right only if it is identical to the reference answer, after minor tidying such as ignoring capital letters., F1 scoreF1 score, n.: A single score that combines how many of a model's alarms are right and how many real cases it catches, and stays low unless both are good., IBM WatsonIBM Watson, prop. n.: An IBM question-answering computer that beat two champions of the US quiz show Jeopardy! in 2011. IBM then sold business AI products under the Watson name, many of which disappointed., lost in the middlelost in the middle, phr.: AI models reading a long input behave a little like a skimmer: they reliably catch facts placed near its start or end but are much likelier to miss the same fact buried halfway through., mean reciprocal rankmean reciprocal rank, n.: A score for search: for each question, note how far down the first correct answer appears, take one over that position, and average across questions. First place scores 1, second place 0.5., natural language understandingnatural language understanding, n.: The part of language AI concerned with working out what a piece of text means — who did what to whom, what the writer wants and what follows from it — rather than with producing text., SQuADSQuAD, prop. n.: A Stanford collection of more than 100,000 questions about Wikipedia articles, each solved by highlighting the words in the article that give the answer; it drove rapid progress in machine reading from 2016., TRECTREC, prop. n.: A yearly contest, run by a US government standards agency since 1992, in which search-engine researchers test their systems on the same documents and questions, with people judging which results are actually relevant., visual question answeringvisual question answering, n.: Asking an AI a question about a picture, such as 'How many people are wearing hats?', and having it answer in words., coreference resolutioncoreference resolution, n.: Working out which words in a text are talking about the same someone or something — for instance, that 'she', 'the senator' and 'Ms. Okafor' all mean one woman., semantic parsingsemantic parsing, n.: Translating a sentence into a precise, machine-executable form — such as a database query or a logical formula — so that a computer can act on what was asked.