instruction tuning
In plain wordsTraining a language model on many tasks written as plain instructions with correct answers, so it learns to do what a request asks instead of just continuing the text.
Supervised fine-tuningsupervised fine-tuning, n.: Teaching a pretrained AI model by showing it many examples of good answers to requests, so it learns to imitate them — like learning a job by studying worked examples. that teaches a pretrained language model to treat a written request as a task to carry out. The training set mixes many different tasks, each example pairing an instruction (with optional input) with the desired output, and the aim is a model that also handles kinds of task it never trained on when given only an instruction (zero-shotzero-shot learning, n.: Doing a task without being shown a single example of it — say, an AI sorting reviews into positive and negative just from being told to, or recognizing an animal it never saw in training.), along with general instruction followinginstruction following, n.: An AI's ability to do exactly what it is asked, including details such as length, format, language and things to avoid, rather than something merely related.. The first systems rewrote existing NLP benchmarks — translation, question answering, classification — as instruction templates; later practice relies on human-written or model-generated conversational data, as in Self-InstructSelf-Instruct, n.: A technique in which a language model writes its own practice tasks and answers, starting from a handful of human examples; the results are filtered and used to train it to follow instructions.. Benefits increase with model size and with the number and variety of training tasks. Instruction tuning is the usual first step in producing an instruction-tuned modelinstruction-tuned model, n.: A language model given extra training on examples of instructions paired with good responses, so it does what you ask instead of merely continuing your text. before preference optimization, and the term is now often used interchangeably with SFT on chat data.
Instruction tuning on 1,800 tasks let the model handle prompts unlike anything it saw in training.
Origin
Named by Jason Wei and colleagues at Google in the FLAN paper (arXiv, September 2021; ICLR 2022), which popularized the method: their 137-billion-parameter model, tuned on more than 60 NLP datasets rephrased as instructions, beat zero-shot GPT-3 on 20 of 25 evaluated datasets. Related 2021 work included Natural Instructions (Swaroop Mishra and colleagues, April) and T0 (Victor Sanh and colleagues, October); Flan-T5 (Hyung Won Chung and colleagues, 2022) scaled the idea to about 1,800 tasks.
Usage note
Not the same as prompt tuningprompt tuning, n.: Teaching a frozen language model a new task by learning a few made-up 'words' — lists of numbers rather than real text — that are placed before the input to steer the model. or prefix tuningprefix tuning, n.: Customizing a frozen language model by learning a short sequence of invisible 'virtual words' that is placed in front of every input at every layer, steering what the model produces., which learn soft prompts and leave the model unchanged. Some authors reserve 'instruction tuning' for multi-task NLP collections and 'SFT' for conversational demonstrations, but the distinction is not consistently observed.
Related
- supervised fine-tuningsupervised fine-tuning, n.: Teaching a pretrained AI model by showing it many examples of good answers to requests, so it learns to imitate them — like learning a job by studying worked examples.
- instruction-tuned modelinstruction-tuned model, n.: A language model given extra training on examples of instructions paired with good responses, so it does what you ask instead of merely continuing your text.
- instruction datasetinstruction dataset, n.: A collection of example requests paired with good answers, used to teach a pretrained language model to follow instructions and act as a helpful assistant.
- instruction followinginstruction following, n.: An AI's ability to do exactly what it is asked, including details such as length, format, language and things to avoid, rather than something merely related.
- Self-InstructSelf-Instruct, n.: A technique in which a language model writes its own practice tasks and answers, starting from a handful of human examples; the results are filtered and used to train it to follow instructions.
- multi-task learningmulti-task learning, n.: Training one model to do several related jobs at once, such as spotting objects and judging distance in the same photo, so that practice on each job can sharpen the others.
- zero-shot learningzero-shot learning, n.: Doing a task without being shown a single example of it — say, an AI sorting reviews into positive and negative just from being told to, or recognizing an animal it never saw in training.
- prompt tuningprompt tuning, n.: Teaching a frozen language model a new task by learning a few made-up 'words' — lists of numbers rather than real text — that are placed before the input to steer the model.
- pretrainingpretraining, n.: The first and usually the most expensive stage of building a modern AI model, in which it learns general patterns from enormous amounts of raw text, images or code before being specialized for particular uses.
Sources
- Wei, J. et al. (2021). Finetuned Language Models Are Zero-Shot Learners. ICLR 2022. arXiv:2109.01652. arxiv.org/abs/2109.01652
- Wang, Y. et al. (2022). Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Tasks. EMNLP 2022. arXiv:2204.07705. arxiv.org/abs/2204.07705
Mentioned in
AI alignmentAI alignment, n.: Making an AI system genuinely try to do what the people it serves want — not a literal-minded, distorted or self-serving version of it. Also the research field working on this., fine-tuningfine-tuning, n.: Taking an AI model that has already learned a lot and training it a little more on a smaller, focused set of examples so it gets better at one job or behaves a certain way., large language modellarge language model, n.: A computer program trained on enormous amounts of text to predict the words that come next. At vast scale, that simple skill lets it chat, answer questions, translate, summarize and write code., AI assistantAI assistant, n.: An AI program, such as ChatGPT, Claude or Gemini, that helps a person through conversation: answering questions, writing, summarizing, coding and, increasingly, taking actions on their behalf., InstructGPTInstructGPT, prop. n.: A 2022 OpenAI language model trained with human feedback to follow instructions; people preferred its answers to those of a model 100 times larger, and its recipe led directly to ChatGPT., instruction datasetinstruction dataset, n.: A collection of example requests paired with good answers, used to teach a pretrained language model to follow instructions and act as a helpful assistant., instruction followinginstruction following, n.: An AI's ability to do exactly what it is asked, including details such as length, format, language and things to avoid, rather than something merely related., instruction-tuned modelinstruction-tuned model, n.: A language model given extra training on examples of instructions paired with good responses, so it does what you ask instead of merely continuing your text., mode collapsemode collapse, n.: When a generator gets stuck producing only a few kinds of output — the same face, the same joke — instead of the full variety it was meant to learn., multi-task learningmulti-task learning, n.: Training one model to do several related jobs at once, such as spotting objects and judging distance in the same photo, so that practice on each job can sharpen the others., supervised fine-tuningsupervised fine-tuning, n.: Teaching a pretrained AI model by showing it many examples of good answers to requests, so it learns to imitate them — like learning a job by studying worked examples., zero-shot learningzero-shot learning, n.: Doing a task without being shown a single example of it — say, an AI sorting reviews into positive and negative just from being told to, or recognizing an animal it never saw in training., prompt tuningprompt tuning, n.: Teaching a frozen language model a new task by learning a few made-up 'words' — lists of numbers rather than real text — that are placed before the input to steer the model., Self-InstructSelf-Instruct, n.: A technique in which a language model writes its own practice tasks and answers, starting from a handful of human examples; the results are filtered and used to train it to follow instructions., superficial alignment hypothesissuperficial alignment hypothesis, n.: The idea that a language model learns nearly all its knowledge and skills from its initial bulk training, and that later tuning mainly teaches it the style and format to use when talking to people., T5T5, prop. n.: A 2019 Google model that treats every language task — translating, summarizing, answering, classifying — as turning one piece of text into another, using one shared design.