Skip to content
Last Dictionary
Plane
AI Engineering & MLOpsAgents & HarnessesData & Datasets

chunking

nounAlso: text splitting, document chunking, chunk size, chunk overlap, semantic chunking, late chunking

In plain wordsCutting long documents into smaller pieces before storing them for AI search, so the system can find and hand the model just the relevant part instead of a whole manual.

  1. 1

    Retrieval

    The step in a retrieval-augmented generationretrieval-augmented generation, n.: Giving an AI an open-book exam: before answering, it looks up relevant passages in a set of documents and bases its reply on them, often citing where each point came from. or search pipeline that splits documents into passages ('chunks') which are embedded, indexed and retrieved as units. Strategies include fixed-size windows measured in tokenstoken, n.: The small chunk of text an AI language model reads and writes — often a word, part of a word or a punctuation mark. AI usage, limits and prices are counted in tokens. with some overlap; recursive splitting on paragraph, sentence and word boundaries; structure-aware splitting by headings, code functions or table rows; semantic chunking, which breaks where the embeddingtext embedding, n.: A list of numbers that captures what a piece of text means, so that texts with similar meanings get similar lists. Computers use them to search, sort and group text by meaning. similarity of adjacent sentences drops; and parent–child schemes that retrieve small chunks but return their surrounding section. Chunk size is a trade-off: small chunks match precisely but lose context, while large ones dilute the embedding and waste the context windowcontext window, n.: How much an AI model can read and keep in view at once: the instructions, the conversation so far and any attached documents. Text that does not fit has to be cut, summarized or left out.. Prefixing each chunk with document-level context (contextual retrievalcontextual retrieval, n.: A fix for searching cut-up documents: before storing each piece, an AI writes a short note saying where it came from and what it is about, so the piece still makes sense on its own.) or embedding the whole document before splitting it (late chunking) reduces lost context.

    Recall improved once we stopped chunking mid-table and split on section headings instead.

  2. 2

    Cognitive architectures

    A learning mechanism, best known from the Soar cognitive architecturecognitive architecture, n.: A blueprint for a complete thinking system — how memory, attention, learning and decision-making fit together — used both to build general AI agents and to model how the human mind works., that caches the result of a problem-solving episode as a new rule (a 'chunk') so the same subproblem is solved directly next time; it echoes the psychological notion of grouping items into larger meaningful units.

Origin

Passage-level retrieval was studied in IR from the 1990s; 'chunking' became the standard name for splitting documents in LLM pipelines around 2023, spread by the text splitters of frameworks such as LangChainLangChain, prop. n.: A popular free software toolkit, and the company behind it, for building applications that connect AI language models to documents, databases, tools and multi-step workflows. and LlamaIndexLlamaIndex, prop. n.: A free software toolkit, and the company behind it, for connecting AI language models to an organization's own documents and data so the models can find and use that information.. The psychological sense dates to George Miller's 1956 work on the limits of short-term memory; John Laird, Paul Rosenbloom and Allen Newell described chunking in Soar in 1986.

Related

See also
  • retrieval-augmented generationretrieval-augmented generation, n.: Giving an AI an open-book exam: before answering, it looks up relevant passages in a set of documents and bases its reply on them, often citing where each point came from.
  • contextual retrievalcontextual retrieval, n.: A fix for searching cut-up documents: before storing each piece, an AI writes a short note saying where it came from and what it is about, so the piece still makes sense on its own.
  • text embeddingtext embedding, n.: A list of numbers that captures what a piece of text means, so that texts with similar meanings get similar lists. Computers use them to search, sort and group text by meaning.
  • context windowcontext window, n.: How much an AI model can read and keep in view at once: the instructions, the conversation so far and any attached documents. Text that does not fit has to be cut, summarized or left out.
  • vector databasevector database, n.: A database built to store embeddings — lists of numbers that represent meaning — and to quickly find the stored items most similar to a new one. It is the memory behind many AI search tools.
  • cognitive architecturecognitive architecture, n.: A blueprint for a complete thinking system — how memory, attention, learning and decision-making fit together — used both to build general AI agents and to model how the human mind works.

Sources

  1. Anthropic (2024). Introducing Contextual Retrieval, 19 September 2024. anthropic.com/news/contextual-retrieval
  2. Laird, J. E., Rosenbloom, P. S. & Newell, A. (1986). Chunking in Soar: The anatomy of a general learning mechanism. Machine Learning 1(1): 11–46.

Mentioned in

retrieval-augmented generationretrieval-augmented generation, n.: Giving an AI an open-book exam: before answering, it looks up relevant passages in a set of documents and bases its reply on them, often citing where each point came from., document understandingdocument understanding, n.: Using AI to read documents such as invoices, forms, contracts and scientific papers the way a person does, taking in layout, tables and charts as well as words, to pull out the information needed., GraphRAGGraphRAG, prop. n.: A way of letting AI answer questions about a big pile of documents by first turning them into a map of people, things and their connections, then using that map to find and summarize answers., action chunkingaction chunking, n.: Having a robot's AI plan a short burst of upcoming movements, say the next second, all at once instead of one tiny motion at a time, which makes motion smoother and mistakes rarer., cache-augmented generationcache-augmented generation, n.: An alternative to search-based AI answering: load the whole knowledge base, if it is small enough, into the model's context once, save the model's processed state, and reuse it for every question., contextual retrievalcontextual retrieval, n.: A fix for searching cut-up documents: before storing each piece, an AI writes a short note saying where it came from and what it is about, so the piece still makes sense on its own., LlamaIndexLlamaIndex, prop. n.: A free software toolkit, and the company behind it, for connecting AI language models to an organization's own documents and data so the models can find and use that information.