lost in the middle
In plain wordsAI models reading a long input behave a little like a skimmer: they reliably catch facts placed near its start or end but are much likelier to miss the same fact buried halfway through.
A positional bias in long-contextlong context, n.: The ability of an AI model to work with very long inputs — whole books, codebases or hours of transcripts — in one go, and the techniques that make this possible. language modelslarge language model, n.: A computer program trained on enormous amounts of text to predict the words that come next. At vast scale, that simple skill lets it chat, answer questions, translate, summarize and write code.: plotted against where the needed information sits in the input, accuracy traces a U, high at the two ends and often much lower in between. In the original multi-document question answeringquestion answering, n.: Systems that reply to people's questions with the right answer, whether by finding it in a passage they are given, searching a database or the web, or drawing on what they already know. experiments, GPT-3.5-Turbo given 20 documents scored about 76% with the answer-bearing document first but about 54% with it in the middle — below its 56% with no documents at all. The pattern resembles the serial-position effect in human recall (primacy and recency). It matters for retrieval-augmented generationretrieval-augmented generation, n.: Giving an AI an open-book exam: before answering, it looks up relevant passages in a set of documents and bases its reply on them, often citing where each point came from. and agents; common responses are retrieving fewer, better passages, placing the most relevant ones at the edges of the prompt, restating key instructions at the end, and training on data in which useful information occurs at varied positions.
Because of lost in the middle, we put the top-ranked passage first and restate the question after the documents.
Origin
Named by Nelson F. Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni and Percy Liang in 'Lost in the Middle: How Language Models Use Long Contexts' (arXiv, July 2023; Transactions of the ACL, 2024), which tested multi-document question answering and key-value retrieval.
Usage note
A position effect, distinct from context rotcontext rot, n.: The way an AI model becomes less reliable as you give it more and more text to read, even when the answer is in there somewhere., the general decline as inputs lengthen. How strong it is varies considerably across models and tasks, so it should be measured for the model in use rather than assumed.
Related
- context rotcontext rot, n.: The way an AI model becomes less reliable as you give it more and more text to read, even when the answer is in there somewhere.
- long contextlong context, n.: The ability of an AI model to work with very long inputs — whole books, codebases or hours of transcripts — in one go, and the techniques that make this possible.
- needle in a haystackneedle in a haystack, n.: A test for AI models that read very long documents: one unrelated sentence is hidden somewhere in a mass of text, and the model is asked about it.
- retrieval-augmented generationretrieval-augmented generation, n.: Giving an AI an open-book exam: before answering, it looks up relevant passages in a set of documents and bases its reply on them, often citing where each point came from.
- rerankingreranking, n.: A second, more careful pass over search results: a fast method grabs a few hundred candidates, then a slower, smarter model reorders them so the best come first.
- context engineeringcontext engineering, n.: Managing an AI model's reading material: choosing, trimming and arranging the instructions, documents, tool results and memories it is given on every turn, so that it has what the task needs and little else.
- attention sinkattention sink, n.: A quirk of language models: they pour a large share of their attention onto the very first word of the text, even when it means nothing, as a place to park attention they do not need.
Sources
- Liu, N. F., Lin, K., Hewitt, J., Paranjape, A., Bevilacqua, M., Petroni, F. & Liang, P. (2024). Lost in the Middle: How Language Models Use Long Contexts. Transactions of the Association for Computational Linguistics 12: 157–173. aclanthology.org/2024.tacl-1.9
Mentioned in
context windowcontext window, n.: How much an AI model can read and keep in view at once: the instructions, the conversation so far and any attached documents. Text that does not fit has to be cut, summarized or left out., retrieval-augmented generationretrieval-augmented generation, n.: Giving an AI an open-book exam: before answering, it looks up relevant passages in a set of documents and bases its reply on them, often citing where each point came from., context rotcontext rot, n.: The way an AI model becomes less reliable as you give it more and more text to read, even when the answer is in there somewhere., needle in a haystackneedle in a haystack, n.: A test for AI models that read very long documents: one unrelated sentence is hidden somewhere in a mass of text, and the model is asked about it., rerankingreranking, n.: A second, more careful pass over search results: a fast method grabs a few hundred candidates, then a slower, smarter model reorders them so the best come first., cache-augmented generationcache-augmented generation, n.: An alternative to search-based AI answering: load the whole knowledge base, if it is small enough, into the model's context once, save the model's processed state, and reuse it for every question.