Skip to content
Last Dictionary
Plane
Language Models & TransformersDeep Learning

byte-level model

nounAlso: byte-level language model, tokenizer-free model, token-free model, character-level model, Byte Latent Transformer, BLT, ByT5

In plain wordsA language model that reads raw bytes — the basic units computers use to store text — instead of tokens from a fixed list, avoiding a separate tokenizer but making inputs much longer.

  1. A language modellanguage model, n.: A system that has learned how likely different sequences of words are, so it can predict the next word in a text and, by doing that repeatedly, write text of its own. that operates directly on UTF-8 bytes (a vocabulary of 256 values plus a few control symbols) rather than on subword tokenstoken, n.: The small chunk of text an AI language model reads and writes — often a word, part of a word or a punctuation mark. AI usage, limits and prices are counted in tokens., removing the learned tokenizertokenizer, n.: The component that splits text into tokens and converts them to numbers before an AI model sees it, then turns the model's output numbers back into text.. Benefits include uniform treatment of every language and script, robustness to misspellings and noise, and direct access to characters, which helps spelling, counting and string manipulation. The cost is sequences several times longer than subword sequences, multiplying attention cost; designs therefore compress bytes inside the network into larger units. ByT5 (2021) showed a byte-level T5T5, prop. n.: A 2019 Google model that treats every language task — translating, summarizing, answering, classifying — as turning one piece of text into another, using one shared design. could be competitive; MegaByte (2023) used fixed-size patches in a two-level model; Meta's Byte Latent Transformer (December 2024) groups bytes into variable-length patches according to the entropy of the next byte, and in the first FLOP-controlled scaling study of byte-level models (up to 8 billion parameters and 4 trillion training bytes) reported matching tokenizer-based models with better inference efficiency and robustness.

    The byte-level model never fails at counting the r's in 'strawberry', but it is slower per word.

Origin

Character-level recurrent language models drew attention with work by Ilya Sutskever, James Martens and Geoffrey Hinton (2011). ByT5, by Linting Xue and colleagues at Google (2021), revived byte-level transformers, and Meta's Byte Latent Transformer (Artidoro Pagnoni and colleagues, 2024) renewed interest in tokenizer-free scaling.

Related

See also
  • tokenizationtokenization, n.: The first step in feeding text to a language model: chopping it into tokens — whole words, word pieces or characters — and turning each into a number the model can work with.
  • tokenizertokenizer, n.: The component that splits text into tokens and converts them to numbers before an AI model sees it, then turns the model's output numbers back into text.
  • byte-pair encodingbyte-pair encoding, n.: A popular way to choose the word pieces an AI breaks text into: start from single characters and repeatedly glue together the pair that occurs most often, until the list reaches the desired size.
  • T5T5, prop. n.: A 2019 Google model that treats every language task — translating, summarizing, answering, classifying — as turning one piece of text into another, using one shared design.
  • language modellanguage model, n.: A system that has learned how likely different sequences of words are, so it can predict the next word in a text and, by doing that repeatedly, write text of its own.
  • tokentoken, n.: The small chunk of text an AI language model reads and writes — often a word, part of a word or a punctuation mark. AI usage, limits and prices are counted in tokens.
Contrast with
  • tokenizertokenizer, n.: The component that splits text into tokens and converts them to numbers before an AI model sees it, then turns the model's output numbers back into text.

Sources

  1. Xue, L., Barua, A., Constant, N., Al-Rfou, R., Narang, S., Kale, M., Roberts, A. & Raffel, C. (2022). ByT5: Towards a Token-Free Future with Pre-trained Byte-to-Byte Models. TACL 10.
  2. Pagnoni, A. et al. (2024). Byte Latent Transformer: Patches Scale Better Than Tokens. arXiv:2412.09871. arxiv.org/abs/2412.09871

Mentioned in

byte-pair encodingbyte-pair encoding, n.: A popular way to choose the word pieces an AI breaks text into: start from single characters and repeatedly glue together the pair that occurs most often, until the list reaches the desired size., tokenizationtokenization, n.: The first step in feeding text to a language model: chopping it into tokens — whole words, word pieces or characters — and turning each into a number the model can work with.