fine-tuning
In plain wordsTaking an AI model that has already learned a lot and training it a little more on a smaller, focused set of examples so it gets better at one job or behaves a certain way.
Continued training of an already trained model on a smaller dataset chosen for a particular task, domain or behavior, starting from its learned weightsweights, n.: The learned numbers inside a neural network that set how strongly each part influences the next. The finished set of weights is, in effect, the trained model itself. rather than from random values. Because the general representations acquired in pretrainingpretraining, n.: The first and usually the most expensive stage of building a modern AI model, in which it learns general patterns from enormous amounts of raw text, images or code before being specialized for particular uses. are kept and merely adjusted, far less data and compute are needed than for training from scratch, which makes fine-tuning the standard way of doing transfer learningtransfer learning, n.: Reusing what a model learned on one task to get a head start on another — like a pianist picking up the organ faster than a beginner. It lets models learn new jobs from little data. with neural networks. Methods vary in the training signal — labeled examples or demonstrations (supervised fine-tuningsupervised fine-tuning, n.: Teaching a pretrained AI model by showing it many examples of good answers to requests, so it learns to imitate them — like learning a job by studying worked examples., instruction tuninginstruction tuning, n.: Training a language model on many tasks written as plain instructions with correct answers, so it learns to do what a request asks instead of just continuing the text.), preferences (reinforcement learning from human feedbackreinforcement learning from human feedback, n.: A way of training AI using people's judgments: humans compare pairs of answers, a scoring model learns their preferences, and the AI is then trained to produce answers that score well., direct preference optimizationdirect preference optimization, n.: A simpler way to teach an AI people's preferences: instead of building a separate scoring model, it trains directly on pairs of better and worse answers, making the better ones more likely.) or graded outcomes (reinforcement fine-tuningreinforcement fine-tuning, n.: Improving a trained AI model by trial and error instead of examples to copy: it tries answers, a grader scores them, and the model shifts toward whatever earned high scores.) — and in how much of the model changes: every parameter (full fine-tuningfull fine-tuning, n.: Fine-tuning in which every one of a model's internal numbers is allowed to change — the most flexible option, but also the most expensive in memory and storage.) or only a small selected subset or added module (parameter-efficient fine-tuningparameter-efficient fine-tuning, n.: A family of tricks for customizing a large AI model by training only a tiny fraction of its numbers, or a small add-on, leaving the rest untouched to save memory and money., such as low-rank adaptationlow-rank adaptation, n.: A cheap way to customize a big AI model: instead of changing all its billions of numbers, you train a small add-on of a few million numbers that slightly adjusts how it behaves.). Typical settings use a learning rate well below the pretraining rate and only a few epochsepoch, n.: One complete pass through all the training examples. Smaller models are often trained for many epochs, seeing each example many times; the largest language models may see most of their data only once.. Risks include overfittingoverfitting, n.: When a model learns its training examples too well — quirks and noise included — so it does great on data it has seen but poorly on anything new, like a student who memorizes answers without understanding. small datasets, catastrophic forgettingcatastrophic forgetting, n.: When a neural network trained on something new gets suddenly and badly worse at what it could already do, because the new training rewrites the internal settings the old skills relied on. of general abilities and erosion of earlier safety training.
We fine-tuned the 8B model on 20,000 support tickets, and it now answers in our house style.
Origin
'Fine-tune' (to make small adjustments) is ordinary English, and its first application to network weights is not established. In an early prominent use, Geoffrey Hinton and colleagues (2006) applied 'fine-tuning' to the backpropagation stage that followed layer-by-layer unsupervised pretraining of deep belief nets and autoencoders. The pretrain-then-fine-tune recipe spread to vision (R-CNN, 2014) and language (ULMFiT, GPT and BERT, 2018), and commercial fine-tuning APIs for large language models appeared from about 2021.
Usage note
Fine-tuning changes a model's weights; prompt engineeringprompt engineering, n.: The skill of writing requests to an AI — picking the words, examples, format and background information it receives — so that it reliably gives the result you want, without retraining it. and in-context learningin-context learning, n.: A language model's ability to pick up a new task from instructions or examples placed in its input, on the spot, without any retraining — like a quick study learning a game's rules from one round. do not. It is one tool of post-trainingpost-training, n.: Everything done to an AI model after its initial bulk training on huge amounts of text: the extra teaching that turns a raw text predictor into a helpful, safe assistant or a careful reasoner., not a synonym for it. Fine-tuning a safety-trained model on even a small number of harmful examples, or on ordinary benign data, can weaken its refusals, a known concern for providers that offer fine-tuning services.
Related
- transfer learningtransfer learning, n.: Reusing what a model learned on one task to get a head start on another — like a pianist picking up the organ faster than a beginner. It lets models learn new jobs from little data.
- pretrainingpretraining, n.: The first and usually the most expensive stage of building a modern AI model, in which it learns general patterns from enormous amounts of raw text, images or code before being specialized for particular uses.
- supervised fine-tuningsupervised fine-tuning, n.: Teaching a pretrained AI model by showing it many examples of good answers to requests, so it learns to imitate them — like learning a job by studying worked examples.
- parameter-efficient fine-tuningparameter-efficient fine-tuning, n.: A family of tricks for customizing a large AI model by training only a tiny fraction of its numbers, or a small add-on, leaving the rest untouched to save memory and money.
- full fine-tuningfull fine-tuning, n.: Fine-tuning in which every one of a model's internal numbers is allowed to change — the most flexible option, but also the most expensive in memory and storage.
- low-rank adaptationlow-rank adaptation, n.: A cheap way to customize a big AI model: instead of changing all its billions of numbers, you train a small add-on of a few million numbers that slightly adjusts how it behaves.
- catastrophic forgettingcatastrophic forgetting, n.: When a neural network trained on something new gets suddenly and badly worse at what it could already do, because the new training rewrites the internal settings the old skills relied on.
- post-trainingpost-training, n.: Everything done to an AI model after its initial bulk training on huge amounts of text: the extra teaching that turns a raw text predictor into a helpful, safe assistant or a careful reasoner.
- pretrainingpretraining, n.: The first and usually the most expensive stage of building a modern AI model, in which it learns general patterns from enormous amounts of raw text, images or code before being specialized for particular uses.
- prompt engineeringprompt engineering, n.: The skill of writing requests to an AI — picking the words, examples, format and background information it receives — so that it reliably gives the result you want, without retraining it.
- in-context learningin-context learning, n.: A language model's ability to pick up a new task from instructions or examples placed in its input, on the spot, without any retraining — like a quick study learning a game's rules from one round.
Sources
- Hinton, G. E. & Salakhutdinov, R. R. (2006). Reducing the Dimensionality of Data with Neural Networks. Science 313(5786), 504–507.
- Howard, J. & Ruder, S. (2018). Universal Language Model Fine-tuning for Text Classification. ACL 2018.
Mentioned in
datasetdataset, n.: An organized collection of examples gathered for a purpose — such as photos with their captions, or questions with answers — used to teach or test an AI system., few-shot learningfew-shot learning, n.: Getting an AI system to handle a new task or recognize a new kind of thing from just a handful of examples, the way a person can learn a new word after hearing it once or twice., open-weight modelopen-weight model, n.: An AI model whose trained internal numbers are published for anyone to download and run on their own computers, even if the data and code used to make it stay private., post-trainingpost-training, n.: Everything done to an AI model after its initial bulk training on huge amounts of text: the extra teaching that turns a raw text predictor into a helpful, safe assistant or a careful reasoner., pretrainingpretraining, n.: The first and usually the most expensive stage of building a modern AI model, in which it learns general patterns from enormous amounts of raw text, images or code before being specialized for particular uses., prompt engineeringprompt engineering, n.: The skill of writing requests to an AI — picking the words, examples, format and background information it receives — so that it reliably gives the result you want, without retraining it., retrieval-augmented generationretrieval-augmented generation, n.: Giving an AI an open-book exam: before answering, it looks up relevant passages in a set of documents and bases its reply on them, often citing where each point came from., trainingtraining, n.: The process by which an AI model learns: it is shown huge numbers of examples and its internal settings are adjusted, bit by bit, until its answers get better., transfer learningtransfer learning, n.: Reusing what a model learned on one task to get a head start on another — like a pianist picking up the organ faster than a beginner. It lets models learn new jobs from little data., activation steeringactivation steering, n.: Changing an AI model's behavior by adjusting the internal numbers it computes while it runs — like turning a hidden dial — instead of rewording the prompt or retraining the model., adapteradapter, n.: A small trainable add-on slotted into a large frozen AI model to teach it a new task or style, like a plug adapter that lets one device work in many countries., adversarial trainingadversarial training, n.: Toughening an AI model by deliberately showing it attacks during training — the tricky inputs that fool it — along with the right responses, so it learns to resist them., agent skillagent skill, n.: A packaged how-to guide for an AI agent — written steps, sometimes with small programs and examples attached — that sits unread until a job of that kind comes up, when the agent opens it., AI engineeringAI engineering, n.: The craft of building useful, reliable products on top of AI models — choosing models, writing prompts, connecting data and tools, testing quality and running it all safely at scale., AI modelAI model, n.: The trained core of an AI: a large file of learned numbers plus the recipe for using them, which turns an input such as a question or a photo into an output., Alec RadfordAlec Radford, prop. n.: An American AI researcher who led OpenAI's first GPT language models and its CLIP and Whisper systems, key steps toward today's chatbots and multimodal AI., backbonebackbone, n.: The main, usually pretrained part of a vision model that turns an image into useful internal features; smaller task-specific parts are attached to it, like limbs to a spine., backdoor attackbackdoor attack, n.: Secretly training an AI model to misbehave whenever it sees a hidden signal — a sticker, a word, a date — while it acts perfectly normally the rest of the time., base modelbase model, n.: A language model fresh from its first broad training on text, before being taught to follow instructions or chat. It continues whatever text it is given instead of answering., BERTBERT, prop. n.: An influential 2018 Google language model that learned to understand text by filling in hidden words. It made search engines and many language tools much better at grasping meaning in context., catastrophic forgettingcatastrophic forgetting, n.: When a neural network trained on something new gets suddenly and badly worse at what it could already do, because the new training rewrites the internal settings the old skills relied on., continued pretrainingcontinued pretraining, n.: Taking an already-trained AI model and resuming its broad, read-everything training on new material — a specialty such as medicine or law, a new language, or fresher data — rather than starting from scratch., domain adaptationdomain adaptation, n.: Adjusting a model trained on one kind of data, such as daytime photos or one hospital's records, so that it still works on a different but related kind, such as night photos or another hospital's., DreamBoothDreamBooth, prop. n.: A way to teach an image generator one specific subject — your dog, your face, a product — from a handful of photos, so it can draw that exact subject in new scenes.