needle in a haystack
In plain wordsA test for AI models that read very long documents: one unrelated sentence is hidden somewhere in a mass of text, and the model is asked about it.
A long contextlong context, n.: The ability of an AI model to work with very long inputs — whole books, codebases or hours of transcripts — in one go, and the techniques that make this possible. retrieval test, named for the idiom: one out-of-place statement, the needle, is buried in a long stretch of unrelated filler text, the haystack, and the model is then asked a question only that statement can answer. Repeating the test while varying both how long the input is and how deep in it the statement sits yields a heatmap of where in its context windowcontext window, n.: How much an AI model can read and keep in view at once: the instructions, the conversation so far and any attached documents. Text that does not fit has to be cut, summarized or left out. a model reliably recalls information. The test exposed recall failures at long lengths and in the middle of documents (compare lost in the middlelost in the middle, phr.: AI models reading a long input behave a little like a skimmer: they reliably catch facts placed near its start or end but are much likelier to miss the same fact buried halfway through.) and became a standard chart in model launch reports. Because leading models soon scored almost perfectly, and because a needle that clashes with its surroundings can be found by surface matching, harder variants followed: multiple needles, needles that must be combined or reasoned over, distractors resembling the needle, and needles sharing no words with the question, as in RULER (2024) and NoLiMa (2025).
It passes needle in a haystack at a million tokens but fails when asked to combine three facts spread through the document.
Origin
Devised by Greg Kamradt in November 2023, who hid a sentence about the best thing to do in San Francisco among Paul Graham's essays to test the newly released long contexts of GPT-4 Turbo (128,000 tokens) and Claude 2.1 (200,000 tokens), publishing his code on GitHub. The name is the old idiom for a hopeless search.
Usage note
Passing shows retrieval of a salient fact, not understanding of a long document; strong scores often coexist with context rotcontext rot, n.: The way an AI model becomes less reliable as you give it more and more text to read, even when the answer is in there somewhere. on realistic tasks. In 2024 a model under test remarked that the planted sentence seemed out of place — an early public example of evaluation awarenessevaluation awareness, n.: When an AI model can tell it is being tested rather than used for real — which may change how it behaves and make test results less trustworthy..
Related
- long contextlong context, n.: The ability of an AI model to work with very long inputs — whole books, codebases or hours of transcripts — in one go, and the techniques that make this possible.
- context windowcontext window, n.: How much an AI model can read and keep in view at once: the instructions, the conversation so far and any attached documents. Text that does not fit has to be cut, summarized or left out.
- lost in the middlelost in the middle, phr.: AI models reading a long input behave a little like a skimmer: they reliably catch facts placed near its start or end but are much likelier to miss the same fact buried halfway through.
- context rotcontext rot, n.: The way an AI model becomes less reliable as you give it more and more text to read, even when the answer is in there somewhere.
- retrieval-augmented generationretrieval-augmented generation, n.: Giving an AI an open-book exam: before answering, it looks up relevant passages in a set of documents and bases its reply on them, often citing where each point came from.
- evaluation awarenessevaluation awareness, n.: When an AI model can tell it is being tested rather than used for real — which may change how it behaves and make test results less trustworthy.
Sources
- Kamradt, G. (2023). LLMTest_NeedleInAHaystack (GitHub repository). github.com/gkamradt/LLMTest_NeedleInAHaystack
- Hsieh, C.-P. et al. (2024). RULER: What's the Real Context Size of Your Long-Context Language Models? COLM 2024.
Mentioned in
context windowcontext window, n.: How much an AI model can read and keep in view at once: the instructions, the conversation so far and any attached documents. Text that does not fit has to be cut, summarized or left out., context rotcontext rot, n.: The way an AI model becomes less reliable as you give it more and more text to read, even when the answer is in there somewhere., long contextlong context, n.: The ability of an AI model to work with very long inputs — whole books, codebases or hours of transcripts — in one go, and the techniques that make this possible., lost in the middlelost in the middle, phr.: AI models reading a long input behave a little like a skimmer: they reliably catch facts placed near its start or end but are much likelier to miss the same fact buried halfway through., recallrecall, n.: Of all the items that really were positive, the share the model managed to find; a cancer screen with high recall misses very few people who actually have cancer.