continued pretraining
In plain wordsTaking an already-trained AI model and resuming its broad, read-everything training on new material — a specialty such as medicine or law, a new language, or fresher data — rather than starting from scratch.
Further training of a pretrained model with its original self-supervised objective — for language models, next-token predictionnext-token prediction, n.: The simple task most AI chatbots were trained on: given some text, guess the next small piece of it. Repeating that guess, piece by piece, produces whole answers. on unlabeled text — on a new corpus, to add domain knowledge, languages, modalities, longer context or more recent information while reusing the original investment. Unlike supervised fine-tuningsupervised fine-tuning, n.: Teaching a pretrained AI model by showing it many examples of good answers to requests, so it learns to imitate them — like learning a job by studying worked examples., it uses raw text, often hundreds of billions of tokens: Meta's Code Llama, for example, continued Llama 2's pretraining on 500 billion tokens of mostly code. The main risks are catastrophic forgettingcatastrophic forgetting, n.: When a neural network trained on something new gets suddenly and badly worse at what it could already do, because the new training rewrites the internal settings the old skills relied on. of general abilities and instability when the learning rate is raised again. Standard remedies are mixing in a fraction of the original data ('replay'), re-warming and re-decaying the learning ratelearning rate, n.: How big a step a model takes each time it adjusts its settings during training. Too large and learning goes haywire; too small and it crawls., and starting from a checkpointcheckpoint, n.: A saved snapshot of a model partway through or at the end of training, like a save point in a video game: training can resume from it after a crash, and finished snapshots are what people download and run. taken before the original decay, which warmup-stable-decay scheduleswarmup-stable-decay schedule, n.: A plan for how fast a model learns during training: speed up briefly, hold steady for most of the run, then slow down sharply at the end — which lets labs stop or extend training without starting over. make convenient.
Continued pretraining on 80 billion tokens of legal text cut perplexity on contracts by a third.
Origin
Early examples include BioBERT (Jinhyuk Lee and colleagues, 2019), which continued BERT's pretraining on biomedical text. Suchin Gururangan and colleagues' 'Don't Stop Pretraining' (ACL 2020) systematized domain- and task-adaptive pretraining, and the practice became routine with open-weight LLMs from 2023.
Usage note
Distinguish from mid-trainingmid-training, n.: A middle stage in building a language model, between its broad initial training and its tuning into an assistant: it keeps learning the same way, but from a smaller, carefully chosen diet of math, code, reasoning or long documents., a planned late stage of the original recipe aimed at broad capabilities, and from continual learningcontinual learning, n.: The ability of an AI to keep learning new things over time from a stream of experience, without forgetting what it already knew and without being retrained from scratch., ongoing learning from a data stream without forgetting. Long-context extension is described under both this name and mid-training. 'Continued' and 'continual' pretraining are used almost interchangeably.
Related
- pretrainingpretraining, n.: The first and usually the most expensive stage of building a modern AI model, in which it learns general patterns from enormous amounts of raw text, images or code before being specialized for particular uses.
- mid-trainingmid-training, n.: A middle stage in building a language model, between its broad initial training and its tuning into an assistant: it keeps learning the same way, but from a smaller, carefully chosen diet of math, code, reasoning or long documents.
- catastrophic forgettingcatastrophic forgetting, n.: When a neural network trained on something new gets suddenly and badly worse at what it could already do, because the new training rewrites the internal settings the old skills relied on.
- fine-tuningfine-tuning, n.: Taking an AI model that has already learned a lot and training it a little more on a smaller, focused set of examples so it gets better at one job or behaves a certain way.
- transfer learningtransfer learning, n.: Reusing what a model learned on one task to get a head start on another — like a pianist picking up the organ faster than a beginner. It lets models learn new jobs from little data.
- warmup-stable-decay schedulewarmup-stable-decay schedule, n.: A plan for how fast a model learns during training: speed up briefly, hold steady for most of the run, then slow down sharply at the end — which lets labs stop or extend training without starting over.
- supervised fine-tuningsupervised fine-tuning, n.: Teaching a pretrained AI model by showing it many examples of good answers to requests, so it learns to imitate them — like learning a job by studying worked examples.
Sources
- Gururangan, S., Marasović, A., Swayamdipta, S., Lo, K., Beltagy, I., Downey, D. & Smith, N. A. (2020). Don't Stop Pretraining: Adapt Language Models to Domains and Tasks. ACL 2020.
- Rozière, B. et al. (2023). Code Llama: Open Foundation Models for Code. arXiv:2308.12950.
Mentioned in
knowledge cutoffknowledge cutoff, n.: The date after which an AI model knows nothing from its training, because its training data stops there. Without a web search tool, it cannot know about later events., post-trainingpost-training, n.: Everything done to an AI model after its initial bulk training on huge amounts of text: the extra teaching that turns a raw text predictor into a helpful, safe assistant or a careful reasoner., pretrainingpretraining, n.: The first and usually the most expensive stage of building a modern AI model, in which it learns general patterns from enormous amounts of raw text, images or code before being specialized for particular uses., checkpointcheckpoint, n.: A saved snapshot of a model partway through or at the end of training, like a save point in a video game: training can resume from it after a crash, and finished snapshots are what people download and run., mid-trainingmid-training, n.: A middle stage in building a language model, between its broad initial training and its tuning into an assistant: it keeps learning the same way, but from a smaller, carefully chosen diet of math, code, reasoning or long documents., context extensioncontext extension, n.: Techniques for stretching how much text a trained AI model can handle at once, usually by adjusting how it represents word positions and then training it briefly on longer examples., sparse upcyclingsparse upcycling, n.: Recycling an already-trained AI model into a larger 'team of specialists' model by copying parts of it, so the new model starts from what the old one learned instead of from scratch., warmup-stable-decay schedulewarmup-stable-decay schedule, n.: A plan for how fast a model learns during training: speed up briefly, hold steady for most of the run, then slow down sharply at the end — which lets labs stop or extend training without starting over.