context extension
In plain wordsTechniques for stretching how much text a trained AI model can handle at once, usually by adjusting how it represents word positions and then training it briefly on longer examples.
Methods for increasing the context windowcontext window, n.: How much an AI model can read and keep in view at once: the instructions, the conversation so far and any attached documents. Text that does not fit has to be cut, summarized or left out. of a pretrained language modellanguage model, n.: A system that has learned how likely different sequences of words are, so it can predict the next word in a text and, by doing that repeatedly, write text of its own. beyond the length it was trained on, far more cheaply than pretraining at the new length. For models using RoPErotary positional embedding, n.: A way of telling a transformer where each word is by rotating parts of its internal numbers by an angle that grows with position, so the model can read off how far apart two words are., the main approaches rescale positions or frequencies so that unseen distances map into the trained range: position interpolation linearly compresses position indices; NTK-aware scaling raises the RoPE base so that high-frequency dimensions barely change while low-frequency ones stretch; YaRN interpolates different frequency bands by different amounts and adjusts attention temperature. These are usually followed by brief continued training on long documents, often within mid-trainingmid-training, n.: A middle stage in building a language model, between its broad initial training and its tuning into an assistant: it keeps learning the same way, but from a smaller, carefully chosen diet of math, code, reasoning or long documents., sometimes in stages (for example 8K, then 32K, then 128K tokens). Models trained with ALiBiALiBi, abbr.: A simple way to give a transformer a sense of word order: attention between two words is penalized more the farther apart they are, which helps it handle texts longer than those it trained on. or without position encodings aim to extrapolate without adjustment. Extended windows need checking on effective-use benchmarks, since nominal length can exceed what a model reliably uses.
After YaRN scaling and 2B tokens of long-document training, the model handles 128K reliably.
Origin
Position interpolation was published by Shouyuan Chen and colleagues at Meta in June 2023 and proposed concurrently by a developer known as kaiokendev; NTK-aware scaling circulated in community posts in mid-2023, and YaRN (Bowen Peng, Jeffrey Quesnelle, Honglu Fan and Enrico Shippole) followed in August 2023.
Related
- context windowcontext window, n.: How much an AI model can read and keep in view at once: the instructions, the conversation so far and any attached documents. Text that does not fit has to be cut, summarized or left out.
- long contextlong context, n.: The ability of an AI model to work with very long inputs — whole books, codebases or hours of transcripts — in one go, and the techniques that make this possible.
- rotary positional embeddingrotary positional embedding, n.: A way of telling a transformer where each word is by rotating parts of its internal numbers by an angle that grows with position, so the model can read off how far apart two words are.
- positional encodingpositional encoding, n.: Extra information added to each word so a transformer knows where it sits in the text. Without it, 'dog bites man' and 'man bites dog' would look the same to the model.
- ALiBiALiBi, abbr.: A simple way to give a transformer a sense of word order: attention between two words is penalized more the farther apart they are, which helps it handle texts longer than those it trained on.
- continued pretrainingcontinued pretraining, n.: Taking an already-trained AI model and resuming its broad, read-everything training on new material — a specialty such as medicine or law, a new language, or fresher data — rather than starting from scratch.
Sources
- Chen, S., Wong, S., Chen, L. & Tian, Y. (2023). Extending Context Window of Large Language Models via Positional Interpolation. arXiv:2306.15595. arxiv.org/abs/2306.15595
- Peng, B., Quesnelle, J., Fan, H. & Shippole, E. (2023). YaRN: Efficient Context Window Extension of Large Language Models. ICLR 2024. arxiv.org/abs/2309.00071
Mentioned in
context windowcontext window, n.: How much an AI model can read and keep in view at once: the instructions, the conversation so far and any attached documents. Text that does not fit has to be cut, summarized or left out., long contextlong context, n.: The ability of an AI model to work with very long inputs — whole books, codebases or hours of transcripts — in one go, and the techniques that make this possible., mid-trainingmid-training, n.: A middle stage in building a language model, between its broad initial training and its tuning into an assistant: it keeps learning the same way, but from a smaller, carefully chosen diet of math, code, reasoning or long documents., positional encodingpositional encoding, n.: Extra information added to each word so a transformer knows where it sits in the text. Without it, 'dog bites man' and 'man bites dog' would look the same to the model., ALiBiALiBi, abbr.: A simple way to give a transformer a sense of word order: attention between two words is penalized more the farther apart they are, which helps it handle texts longer than those it trained on., context parallelismcontext parallelism, n.: A way of training or running a model on extremely long inputs — whole books or codebases — by splitting the text across many chips and having them swap the information each part needs to 'see' the others., rotary positional embeddingrotary positional embedding, n.: A way of telling a transformer where each word is by rotating parts of its internal numbers by an angle that grows with position, so the model can read off how far apart two words are.