mid-training

A training stage between pretrainingpretraining, n.: The first and usually the most expensive stage of building a modern AI model, in which it learns general patterns from enormous amounts of raw text, images or code before being specialized for particular uses. and post-trainingpost-training, n.: Everything done to an AI model after its initial bulk training on huge amounts of text: the extra teaching that turns a raw text predictor into a helpful, safe assistant or a careful reasoner. in which a pretrained model keeps the same self-supervised objective, usually next-token predictionnext-token prediction, n.: The simple task most AI chatbots were trained on: given some text, guess the next small piece of it. Repeating that guess, piece by piece, produces whole answers., but trains on a smaller, deliberately reweighted data mixturedata mixture, n.: The recipe of different kinds of data a model is trained on — how much web text, code, books, math, science and other languages — which strongly shapes what the finished model is good at. to build particular capabilities without losing general ones: upweighted mathematics, code and science; synthetic and reasoning-style data; instruction- or chat-formattedchat template, n.: The exact text layout a chat AI was trained on for marking who is speaking — system, user or assistant. Software must reproduce it precisely, or the model performs noticeably worse. text that prepares for supervised fine-tuningsupervised fine-tuning, n.: Teaching a pretrained AI model by showing it many examples of good answers to requests, so it learns to imitate them — like learning a job by studying worked examples.; and long documents for context extensioncontext extension, n.: Techniques for stretching how much text a trained AI model can handle at once, usually by adjusting how it represents word positions and then training it briefly on longer examples.. It usually coincides with the learning-rate decay (annealingannealing, n.: The last stretch of training, when the model's learning is slowed to a stop — often while it is fed a small helping of the best data — so it settles into its final, sharpest version.), typically uses a few percent to about a tenth of training compute, and still yields a base modelbase model, n.: A language model fresh from its first broad training on text, before being taught to follow instructions or chat. It continues whatever text it is given instead of answering.. It differs from post-training, which shapes behavior with curated demonstrations, preferences or rewards, and from continued pretrainingcontinued pretraining, n.: Taking an already-trained AI model and resuming its broad, read-everything training on new material — a specialty such as medicine or law, a new language, or fresher data — rather than starting from scratch., which adapts an already finished model to a new domain, language or period, often outside the original developer's recipe. Boundaries are fuzzy: one lab's mid-training is another's late pretraining, and some pipelines use it mainly to teach chat and tool-use formats. Studies in 2025 linked mid-training on reasoning data to more effective later reinforcement learningreinforcement learning with verifiable rewards, n.: Training an AI by letting it attempt problems whose answers can be checked automatically — a math result, a passing program — and rewarding it only when the check succeeds..

Mid-training on 300 billion tokens of math and code lifted the base model's GSM8K score by twelve points.