byte-pair encoding
In plain wordsA popular way to choose the word pieces an AI breaks text into: start from single characters and repeatedly glue together the pair that occurs most often, until the list reaches the desired size.
A subword tokenizationtokenization, n.: The first step in feeding text to a language model: chopping it into tokens — whole words, word pieces or characters — and turning each into a number the model can work with. algorithm that builds a vocabularyvocabulary, n.: The fixed list of all the tokens a language model knows, each with its own number. Everything the model reads or writes must be built from pieces on this list. by starting from individual characters (or bytes) and repeatedly merging the most frequent adjacent pair of symbols in a training corpus into a new symbol, recording each merge, until a target vocabulary size is reached. New text is tokenized by replaying the learned merges in order. Frequent words thus become single tokens and rare words split into familiar pieces, so out-of-vocabulary words become rare (and, with byte-level BPE, impossible). Byte-level BPE, introduced with GPT-2GPT-2, prop. n.: A 2019 OpenAI language model that wrote surprisingly fluent paragraphs. OpenAI at first withheld the full version, saying it could be misused for fake news, which made it famous., runs on UTF-8 bytes rather than Unicode characters, so the base alphabet has only 256 symbols and any string can be encoded; combined with regex pre-splitting, it is used by the GPT series, Llama 3 and many other LLMslarge language model, n.: A computer program trained on enormous amounts of text to predict the words that come next. At vast scale, that simple skill lets it chat, answer questions, translate, summarize and write code.. Standard BPE is greedy and deterministic; BPE-dropout randomly skips merges during training as a regularizer.
We trained a BPE tokenizer with 64,000 merges on the multilingual mix.
Origin
Philip Gage described byte pair encoding in 1994 as a data-compression method that replaces frequent byte pairs with unused byte values. Rico Sennrich, Barry Haddow and Alexandra Birch adapted it to subword segmentation for neural machine translation (arXiv 2015; ACL 2016), and OpenAI's GPT-2 (2019) introduced the byte-level variant.
Usage note
In compression, BPE shrinks data by substituting unused byte codes; in NLP it grows a vocabulary. The shared name reflects the shared merge-the-most-frequent-pair idea.
Related
- tokenizationtokenization, n.: The first step in feeding text to a language model: chopping it into tokens — whole words, word pieces or characters — and turning each into a number the model can work with.
- tokenizertokenizer, n.: The component that splits text into tokens and converts them to numbers before an AI model sees it, then turns the model's output numbers back into text.
- WordPieceWordPiece, n.: Google's method for splitting words into smaller known pieces, used by BERT. An unfamiliar word such as 'unhappiness' might be broken into parts like 'un', '##happi' and '##ness'.
- SentencePieceSentencePiece, prop. n.: Free software from Google that learns how to split text into tokens straight from raw sentences in any language, treating spaces as ordinary characters, so no language-specific word splitting is needed.
- vocabularyvocabulary, n.: The fixed list of all the tokens a language model knows, each with its own number. Everything the model reads or writes must be built from pieces on this list.
- byte-level modelbyte-level model, n.: A language model that reads raw bytes — the basic units computers use to store text — instead of tokens from a fixed list, avoiding a separate tokenizer but making inputs much longer.
- WordPieceWordPiece, n.: Google's method for splitting words into smaller known pieces, used by BERT. An unfamiliar word such as 'unhappiness' might be broken into parts like 'un', '##happi' and '##ness'.
Sources
- Gage, P. (1994). A New Algorithm for Data Compression. The C Users Journal 12(2).
- Sennrich, R., Haddow, B. & Birch, A. (2016). Neural Machine Translation of Rare Words with Subword Units. ACL 2016. arxiv.org/abs/1508.07909
Mentioned in
tokentoken, n.: The small chunk of text an AI language model reads and writes — often a word, part of a word or a punctuation mark. AI usage, limits and prices are counted in tokens., strawberry problemstrawberry problem, n.: The once-famous failure of AI chatbots to count the r's in 'strawberry' correctly — a simple question that exposed how these models see words in chunks rather than letters., tokenizationtokenization, n.: The first step in feeding text to a language model: chopping it into tokens — whole words, word pieces or characters — and turning each into a number the model can work with., tokenizertokenizer, n.: The component that splits text into tokens and converts them to numbers before an AI model sees it, then turns the model's output numbers back into text., vocabularyvocabulary, n.: The fixed list of all the tokens a language model knows, each with its own number. Everything the model reads or writes must be built from pieces on this list., byte-level modelbyte-level model, n.: A language model that reads raw bytes — the basic units computers use to store text — instead of tokens from a fixed list, avoiding a separate tokenizer but making inputs much longer., SentencePieceSentencePiece, prop. n.: Free software from Google that learns how to split text into tokens straight from raw sentences in any language, treating spaces as ordinary characters, so no language-specific word splitting is needed., WordPieceWordPiece, n.: Google's method for splitting words into smaller known pieces, used by BERT. An unfamiliar word such as 'unhappiness' might be broken into parts like 'un', '##happi' and '##ness'., Zipf's lawZipf's law, n.: The pattern that word use is lopsided: in a big text the top word is roughly twice as common as the runner-up and ten times as common as the tenth, so a few words dominate and most are rare.