teacher forcing
In plain wordsTraining a model that writes one word at a time by always feeding it the correct previous words, rather than its own possibly mistaken guesses, while it learns to predict the next one.
A training procedure for sequence models that predict one element at a time, in which the input at each position is the ground-truth previous element from the training data rather than the model's own previous prediction. Because all targets are known in advance, the per-position losses do not depend on one another's outputs, so a transformertransformer, n.: The design behind today's chatbots and much of modern AI. It handles text by letting every word look at every other word and decide which of them matter for understanding it. can compute them for a whole sequence in a single parallel forward passforward pass, n.: The step where a neural network takes an input and runs it through its layers, one after another, to produce an output — its answer or prediction. using a causal maskcausal mask, n.: A rule inside text-generating AI that stops each word from seeing the words after it, so the model learns to predict what comes next without being able to cheat.; this is how next-token predictionnext-token prediction, n.: The simple task most AI chatbots were trained on: given some text, guess the next small piece of it. Repeating that guess, piece by piece, produces whole answers. pretraining of large language modelslarge language model, n.: A computer program trained on enormous amounts of text to predict the words that come next. At vast scale, that simple skill lets it chat, answer questions, translate, summarize and write code. and supervised fine-tuningsupervised fine-tuning, n.: Teaching a pretrained AI model by showing it many examples of good answers to requests, so it learns to imitate them — like learning a job by studying worked examples. work. Teacher forcing gives stable, efficient maximum-likelihood training, but the model never practices recovering from its own errors, creating a mismatch between training and generation known as exposure biasexposure bias, n.: A weakness of models trained only on perfect examples: when they write on their own, one early mistake puts them in unfamiliar territory they never practiced, and errors can snowball.. Remedies include scheduled sampling, professor forcing, sequence-level training and, for language models, reinforcement learningreinforcement learning, n.: A way of teaching a computer program by trial and error: it tries actions, earns points for good results and penalties for bad ones, and gradually learns which choices pay off over time. on the model's own samples.
Training loss looks great under teacher forcing, but free-running generations drift after a few sentences.
Origin
The term is credited to Ronald J. Williams and David Zipser, who used it in 'A Learning Algorithm for Continually Running Fully Recurrent Neural Networks' (Neural Computation, 1989): replacing a unit's output with the teacher signal during training sped up and stabilized learning in recurrent networks. The practice of feeding target values back as inputs was already in use at the time.
Related
- exposure biasexposure bias, n.: A weakness of models trained only on perfect examples: when they write on their own, one early mistake puts them in unfamiliar territory they never practiced, and errors can snowball.
- next-token predictionnext-token prediction, n.: The simple task most AI chatbots were trained on: given some text, guess the next small piece of it. Repeating that guess, piece by piece, produces whole answers.
- recurrent neural networkrecurrent neural network, n.: A neural network that reads a sequence, such as a sentence or an audio clip, one piece at a time, carrying a running summary forward so that what came earlier can shape how later pieces are understood.
- autoregressive modelautoregressive model, n.: A model that produces a sequence one piece at a time, each new piece depending on the pieces already produced — like writing a sentence word by word without going back.
- causal maskcausal mask, n.: A rule inside text-generating AI that stops each word from seeing the words after it, so the model learns to predict what comes next without being able to cheat.
- sequence-to-sequence modelsequence-to-sequence model, n.: A model that turns one sequence into another, such as a French sentence into English or speech into text, by first reading the whole input and then writing the output step by step.
Sources
- Williams, R. J. & Zipser, D. (1989). A Learning Algorithm for Continually Running Fully Recurrent Neural Networks. Neural Computation 1(2), 270–280.
Mentioned in
sequence-to-sequence modelsequence-to-sequence model, n.: A model that turns one sequence into another, such as a French sentence into English or speech into text, by first reading the whole input and then writing the output step by step., exposure biasexposure bias, n.: A weakness of models trained only on perfect examples: when they write on their own, one early mistake puts them in unfamiliar territory they never practiced, and errors can snowball., on-policy distillationon-policy distillation, n.: A way of teaching a small AI from a bigger one where the small AI writes its own answers and the bigger one grades every word, so the learner is corrected on its own mistakes.