model soup
In plain wordsA single model made by averaging the internal numbers of several copies of a model that were fine-tuned in slightly different ways, often doing better than any one of them.
A single network formed by taking the element-wise mean of the weightsweights, n.: The learned numbers inside a neural network that set how strongly each part influences the next. The finished set of weights is, in effect, the trained model itself. of several fine-tuned versions of one pretrained checkpointcheckpoint, n.: A saved snapshot of a model partway through or at the end of training, like a save point in a video game: training can resume from it after a crash, and finished snapshots are what people download and run., runs that differed in hyperparametershyperparameter, n.: A setting chosen by people before training, such as how fast a model learns or how big it is, as opposed to the internal values the model learns by itself from data., data order or augmentation. A uniform soup averages all ingredients; a greedy soup adds them one at a time, in order of validation accuracy, keeping each only if it helps. It works because fine-tunes of one pretrained model tend to land in the same wide, low-loss basin, so their mean is itself a good solution that often beats every ingredient, especially under distribution shiftdistribution shift, n.: When the data an AI system meets in real use differs from the data it was trained on — new users, places, times or habits — often making it perform worse than tests suggested.; and unlike an output ensembleensemble learning, n.: Combining the predictions of several models — by voting or averaging — so their individual mistakes partly cancel out, giving more accurate and stable results than any one model., it runs at the cost of a single model. Rather than keeping the best run of a hyperparameter sweep and throwing the rest away, a soup puts them all to use; one made from ViT-G models set an ImageNet record of 90.94 percent top-1 accuracy. Model soups are the simplest case of model mergingmodel merging, n.: Combining two or more trained AI models into one by blending their internal numbers directly, so the result shares their skills without any further training..
Instead of keeping only the best of our twelve fine-tuning runs, we made a greedy soup and gained a point.
Origin
Introduced by Mitchell Wortsman, Gabriel Ilharco and colleagues in March 2022 and published at ICML 2022. The culinary name plays on combining ingredients; the method built on earlier evidence that fine-tuned models are linearly connected by low-loss paths.
Usage note
Soups fail when ingredients are fine-tuned too far apart or from different initializations, because their average leaves the shared basin.
Related
- model mergingmodel merging, n.: Combining two or more trained AI models into one by blending their internal numbers directly, so the result shares their skills without any further training.
- task arithmetictask arithmetic, n.: Editing an AI model by adding or subtracting the changes that fine-tuning made to it, as if skills were arrows that can be added together or reversed.
- ensemble learningensemble learning, n.: Combining the predictions of several models — by voting or averaging — so their individual mistakes partly cancel out, giving more accurate and stable results than any one model.
- exponential moving average of weightsexponential moving average of weights, n.: A smoothed copy of a model kept during training, which each update nudges only slightly toward the latest weights. The calmer copy often performs better than the jittery original.
- loss landscapeloss landscape, n.: A way of picturing training as a journey across hilly terrain, where position stands for the model's settings and height for its error. Training searches for low valleys.
- fine-tuningfine-tuning, n.: Taking an AI model that has already learned a lot and training it a little more on a smaller, focused set of examples so it gets better at one job or behaves a certain way.
- ensemble learningensemble learning, n.: Combining the predictions of several models — by voting or averaging — so their individual mistakes partly cancel out, giving more accurate and stable results than any one model.
Sources
- Wortsman, M. et al. (2022). Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time. ICML 2022. arXiv:2203.05482. arxiv.org/abs/2203.05482
Mentioned in
model mergingmodel merging, n.: Combining two or more trained AI models into one by blending their internal numbers directly, so the result shares their skills without any further training., task arithmetictask arithmetic, n.: Editing an AI model by adding or subtracting the changes that fine-tuning made to it, as if skills were arrows that can be added together or reversed.