post-training
In plain wordsEverything done to an AI model after its initial bulk training on huge amounts of text: the extra teaching that turns a raw text predictor into a helpful, safe assistant or a careful reasoner.
The collective name for the training stages applied to a large language modellarge language model, n.: A computer program trained on enormous amounts of text to predict the words that come next. At vast scale, that simple skill lets it chat, answer questions, translate, summarize and write code. after pretrainingpretraining, n.: The first and usually the most expensive stage of building a modern AI model, in which it learns general patterns from enormous amounts of raw text, images or code before being specialized for particular uses., which turn a base modelbase model, n.: A language model fresh from its first broad training on text, before being taught to follow instructions or chat. It continues whatever text it is given instead of answering. that merely continues text into a deployable assistant. A typical pipeline runs supervised fine-tuningsupervised fine-tuning, n.: Teaching a pretrained AI model by showing it many examples of good answers to requests, so it learns to imitate them — like learning a job by studying worked examples. on demonstrations, then preference optimization — reinforcement learning from human feedbackreinforcement learning from human feedback, n.: A way of training AI using people's judgments: humans compare pairs of answers, a scoring model learns their preferences, and the AI is then trained to produce answers that score well., direct preference optimizationdirect preference optimization, n.: A simpler way to teach an AI people's preferences: instead of building a separate scoring model, it trains directly on pairs of better and worse answers, making the better ones more likely. or reinforcement learning from AI feedbackreinforcement learning from AI feedback, n.: Training an AI with preference judgments made by another AI instead of by people, which is cheaper and faster but only as good as the AI doing the judging. — and, since 2024, large-scale reinforcement learning with verifiable rewardsreinforcement learning with verifiable rewards, n.: Training an AI by letting it attempt problems whose answers can be checked automatically — a math result, a passing program — and rewarding it only when the check succeeds. to teach reasoning; knowledge distillationknowledge distillation, n.: Training a smaller or cheaper 'student' AI to copy the behavior of a larger 'teacher' AI, so the student keeps much of the teacher's skill while being faster to run., model mergingmodel merging, n.: Combining two or more trained AI models into one by blending their internal numbers directly, so the result shares their skills without any further training. and safety training are often interleaved. Post-training shapes format, tone, instruction followinginstruction following, n.: An AI's ability to do exactly what it is asked, including details such as length, format, language and things to avoid, rather than something merely related., refusals and tool use, and increasingly builds capabilities: by 2025 laboratories described order-of-magnitude increases in reinforcement-learning compute between model generations, and xAI said it had run Grok 4's reinforcement learning at pretraining scale. It still uses far less data than pretraining, and the boundary is blurred by mid-trainingmid-training, n.: A middle stage in building a language model, between its broad initial training and its tuning into an assistant: it keeps learning the same way, but from a smaller, carefully chosen diet of math, code, reasoning or long documents. and continued pretrainingcontinued pretraining, n.: Taking an already-trained AI model and resuming its broad, read-everything training on new material — a specialty such as medicine or law, a new language, or fresher data — rather than starting from scratch..
The base model is unchanged; all of the gains in tool use came from post-training.
Origin
Earlier, 'post-training' mostly modified other terms, as in post-training quantization, although Hu Xu and colleagues (2019) used 'BERT post-training' for extra domain training of a pretrained model. As the umbrella name for the alignment stages of language-model development it appears by 2023 in OpenAI's GPT-4 Technical Report, which credited a post-training alignment process with better factuality, and it became standard in 2024 through reports such as Meta's Llama 3 paper and the Allen Institute for AI's Tülu 3, an open post-training recipe.
Usage note
Not to be confused with post-training quantizationpost-training quantization, n.: Shrinking an already-trained AI model by storing its numbers in fewer bits, without training it again; a small sample of example inputs is usually enough to fine-tune the rounding., which compresses a finished model without further learning. Some writers use 'alignment' or 'fine-tuning' loosely for all of post-training; in current usage post-training is the umbrella and fine-tuning one of its tools.
Related
- pretrainingpretraining, n.: The first and usually the most expensive stage of building a modern AI model, in which it learns general patterns from enormous amounts of raw text, images or code before being specialized for particular uses.
- fine-tuningfine-tuning, n.: Taking an AI model that has already learned a lot and training it a little more on a smaller, focused set of examples so it gets better at one job or behaves a certain way.
- supervised fine-tuningsupervised fine-tuning, n.: Teaching a pretrained AI model by showing it many examples of good answers to requests, so it learns to imitate them — like learning a job by studying worked examples.
- reinforcement learning from human feedbackreinforcement learning from human feedback, n.: A way of training AI using people's judgments: humans compare pairs of answers, a scoring model learns their preferences, and the AI is then trained to produce answers that score well.
- reinforcement learning with verifiable rewardsreinforcement learning with verifiable rewards, n.: Training an AI by letting it attempt problems whose answers can be checked automatically — a math result, a passing program — and rewarding it only when the check succeeds.
- instruction-tuned modelinstruction-tuned model, n.: A language model given extra training on examples of instructions paired with good responses, so it does what you ask instead of merely continuing your text.
- mid-trainingmid-training, n.: A middle stage in building a language model, between its broad initial training and its tuning into an assistant: it keeps learning the same way, but from a smaller, carefully chosen diet of math, code, reasoning or long documents.
- pretrainingpretraining, n.: The first and usually the most expensive stage of building a modern AI model, in which it learns general patterns from enormous amounts of raw text, images or code before being specialized for particular uses.
- post-training quantizationpost-training quantization, n.: Shrinking an already-trained AI model by storing its numbers in fewer bits, without training it again; a small sample of example inputs is usually enough to fine-tune the rounding.
Sources
- OpenAI (2023). GPT-4 Technical Report. arXiv:2303.08774. arxiv.org/abs/2303.08774
- Lambert, N. et al. (2024). Tülu 3: Pushing Frontiers in Open Language Model Post-Training. arXiv:2411.15124. arxiv.org/abs/2411.15124
Mentioned in
AI alignmentAI alignment, n.: Making an AI system genuinely try to do what the people it serves want — not a literal-minded, distorted or self-serving version of it. Also the research field working on this., fine-tuningfine-tuning, n.: Taking an AI model that has already learned a lot and training it a little more on a smaller, focused set of examples so it gets better at one job or behaves a certain way., large language modellarge language model, n.: A computer program trained on enormous amounts of text to predict the words that come next. At vast scale, that simple skill lets it chat, answer questions, translate, summarize and write code., pretrainingpretraining, n.: The first and usually the most expensive stage of building a modern AI model, in which it learns general patterns from enormous amounts of raw text, images or code before being specialized for particular uses., reinforcement learningreinforcement learning, n.: A way of teaching a computer program by trial and error: it tries actions, earns points for good results and penalties for bad ones, and gradually learns which choices pay off over time., reinforcement learning from human feedbackreinforcement learning from human feedback, n.: A way of training AI using people's judgments: humans compare pairs of answers, a scoring model learns their preferences, and the AI is then trained to produce answers that score well., trainingtraining, n.: The process by which an AI model learns: it is shown huge numbers of examples and its internal settings are adjusted, bit by bit, until its answers get better., AI assistantAI assistant, n.: An AI program, such as ChatGPT, Claude or Gemini, that helps a person through conversation: answering questions, writing, summarizing, coding and, increasingly, taking actions on their behalf., base modelbase model, n.: A language model fresh from its first broad training on text, before being taught to follow instructions or chat. It continues whatever text it is given instead of answering., compute budgetcompute budget, n.: The total amount of computing a team can spend on training (or running) a model, set in advance like a money budget and used to decide how big the model should be and how much data it sees., data labelingdata labeling, n.: The work of attaching correct answers to examples — often done by large paid workforces, sometimes by experts — so that AI systems can learn from them or be tested against them., deep reinforcement learningdeep reinforcement learning, n.: Reinforcement learning in which a large neural network does the learning, letting an AI learn tasks such as video games or robot control directly from raw inputs like screen pixels or camera images., delvedelve, v.: An ordinary English verb meaning to dig into a subject, which AI chatbots used so often in 2023–2024 that it became a joke and a supposed telltale sign of machine-written text., fancy autocompletefancy autocomplete, n.: A dismissive description of AI chatbots as nothing more than a souped-up version of the feature on a phone keyboard that suggests your next word., full fine-tuningfull fine-tuning, n.: Fine-tuning in which every one of a model's internal numbers is allowed to change — the most flexible option, but also the most expensive in memory and storage., LLM-ismsLLM-isms, n.: Words, phrases and habits of style that AI chatbots overuse — certain fancy verbs, tidy lists of three, 'it's not just X, it's Y' — which make writing recognizable as machine-made., mid-trainingmid-training, n.: A middle stage in building a language model, between its broad initial training and its tuning into an assistant: it keeps learning the same way, but from a smaller, carefully chosen diet of math, code, reasoning or long documents., reinforcement learning from AI feedbackreinforcement learning from AI feedback, n.: Training an AI with preference judgments made by another AI instead of by people, which is cheaper and faster but only as good as the AI doing the judging., RL environmentRL environment, n.: A practice world built for training an AI by trial and error: it hands the AI tasks and tools, lets it act, and scores the results so the AI can learn what works., RL scalingRL scaling, n.: The practice, and the finding behind it, of giving AI models far more trial-and-error practice after their initial training, especially on problems with checkable answers, which has so far improved their reasoning fairly predictably., safety trainingsafety training, n.: The part of an AI's training that teaches it to avoid harmful behavior — declining dangerous requests, resisting manipulation and following its developer's rules — on top of being generally helpful., shoggothshoggoth, n.: A tentacled monster from H. P. Lovecraft's horror fiction, used as a meme for large AI models: a vast, alien system hidden behind a small, friendly mask that training adds to make it polite., supervised fine-tuningsupervised fine-tuning, n.: Teaching a pretrained AI model by showing it many examples of good answers to requests, so it learns to imitate them — like learning a job by studying worked examples., alignment taxalignment tax, n.: The price of caution in AI: whatever performance, money or time a developer sacrifices to make a system safe and aligned, compared with an otherwise identical system built with no such care.