vocabulary
In plain wordsThe fixed list of all the tokens a language model knows, each with its own number. Everything the model reads or writes must be built from pieces on this list.
The finite set of tokenstoken, n.: The small chunk of text an AI language model reads and writes — often a word, part of a word or a punctuation mark. AI usage, limits and prices are counted in tokens. that a tokenizertokenizer, n.: The component that splits text into tokens and converts them to numbers before an AI model sees it, then turns the model's output numbers back into text. can produce and a language modellanguage model, n.: A system that has learned how likely different sequences of words are, so it can predict the next word in a text and, by doing that repeatedly, write text of its own. can read and emit, each assigned an integer ID. Its size V fixes the number of rows of the input embeddingembedding, n.: A list of numbers that stands for a word, picture or other item, arranged so that similar items get similar lists. It lets a computer measure similarity of meaning as distance. matrix and the width of the output layer that produces logitslogit, n.: A raw score that a model gives each possible answer before the scores are turned into probabilities. In statistics, the 'log-odds': a probability rewritten on a scale running from minus to plus infinity. over the next token. Larger vocabularies encode text in fewer tokens — saving context and compute per character, especially for non-English text — but add parameters (V × d per embedding matrix) and give rare tokens fewer training examples. Sizes range from about 30,000 (BERTBERT, prop. n.: An influential 2018 Google language model that learned to understand text by filling in hidden words. It made search engines and many language tools much better at grasping meaning in context.) and 50,257 (GPT-2) to roughly 100,000–260,000 in many current models. Vocabularies include special tokensspecial token, n.: A reserved symbol in an AI model's vocabulary that is not ordinary text but a signal — 'start of message', 'end of answer', 'begin thinking' — used to structure what the model reads and writes. for control. Entries that were common in the tokenizer's training text but rare in the model's can stay nearly untrained, producing glitch tokensglitch token, n.: A word fragment in an AI model's vocabulary that the model almost never saw during training, so asking about it can produce bizarre, evasive or garbled answers. that trigger erratic behavior.
Doubling the vocabulary cut sequence lengths by 15% on our Korean data.
Origin
In word-level NLP, the vocabulary was the set of word types kept for a model, with all others mapped to an unknown-word symbol, so out-of-vocabulary words were a chronic problem. Subword methods from 2015–2016 and byte-level vocabularies (2019) made fixed but open-ended vocabularies the norm.
Usage note
Glitch tokens such as ' SolidGoldMagikarp', reported in GPT-2/GPT-3 tokenizers in 2023, became a well-known illustration of under-trained vocabulary entries.
Related
- tokentoken, n.: The small chunk of text an AI language model reads and writes — often a word, part of a word or a punctuation mark. AI usage, limits and prices are counted in tokens.
- tokenizertokenizer, n.: The component that splits text into tokens and converts them to numbers before an AI model sees it, then turns the model's output numbers back into text.
- tokenizationtokenization, n.: The first step in feeding text to a language model: chopping it into tokens — whole words, word pieces or characters — and turning each into a number the model can work with.
- embeddingembedding, n.: A list of numbers that stands for a word, picture or other item, arranged so that similar items get similar lists. It lets a computer measure similarity of meaning as distance.
- unembeddingunembedding, n.: The last step of a language model, which turns its internal state into a score for every token in its vocabulary; the next token is then chosen from those scores.
- special tokenspecial token, n.: A reserved symbol in an AI model's vocabulary that is not ordinary text but a signal — 'start of message', 'end of answer', 'begin thinking' — used to structure what the model reads and writes.
- byte-pair encodingbyte-pair encoding, n.: A popular way to choose the word pieces an AI breaks text into: start from single characters and repeatedly glue together the pair that occurs most often, until the list reaches the desired size.
Mentioned in
tokentoken, n.: The small chunk of text an AI language model reads and writes — often a word, part of a word or a punctuation mark. AI usage, limits and prices are counted in tokens., byte-pair encodingbyte-pair encoding, n.: A popular way to choose the word pieces an AI breaks text into: start from single characters and repeatedly glue together the pair that occurs most often, until the list reaches the desired size., decodingdecoding, n.: The method a language model uses to turn its predictions into actual text — for example, always picking the most likely next word, or choosing somewhat at random among the likely ones., one-hot encodingone-hot encoding, n.: Turning a category such as 'red', 'green' or 'blue' into a row of zeros with a single one marking which category it is, so that a model can use it as numbers., special tokenspecial token, n.: A reserved symbol in an AI model's vocabulary that is not ordinary text but a signal — 'start of message', 'end of answer', 'begin thinking' — used to structure what the model reads and writes., tokenizationtokenization, n.: The first step in feeding text to a language model: chopping it into tokens — whole words, word pieces or characters — and turning each into a number the model can work with., tokenizertokenizer, n.: The component that splits text into tokens and converts them to numbers before an AI model sees it, then turns the model's output numbers back into text., glitch tokenglitch token, n.: A word fragment in an AI model's vocabulary that the model almost never saw during training, so asking about it can produce bizarre, evasive or garbled answers., SentencePieceSentencePiece, prop. n.: Free software from Google that learns how to split text into tokens straight from raw sentences in any language, treating spaces as ordinary characters, so no language-specific word splitting is needed., unembeddingunembedding, n.: The last step of a language model, which turns its internal state into a score for every token in its vocabulary; the next token is then chosen from those scores., WordPieceWordPiece, n.: Google's method for splitting words into smaller known pieces, used by BERT. An unfamiliar word such as 'unhappiness' might be broken into parts like 'un', '##happi' and '##ness'., Zipf's lawZipf's law, n.: The pattern that word use is lopsided: in a big text the top word is roughly twice as common as the runner-up and ten times as common as the tenth, so a few words dominate and most are rare.