data augmentation
In plain wordsCreating extra training examples by changing existing ones in ways that keep their meaning — a flipped, cropped or recolored photo of a cat still shows a cat — so the model learns more robustly.
The practice of enlarging the effective training datatraining data, n.: The examples an AI system learns from. Whatever patterns, skills, gaps and biases are in the training data tend to show up in the finished system. by applying label-preserving transformations to existing examples, usually at random and on the fly during training. In vision, standard augmentations include random crops, flips, rotations, scaling, color jitter, blur and erasing patches; mixing methods such as mixup and CutMix blend pairs of images and their labels; searched or randomized policies (AutoAugment, RandAugment) choose transformation types and strengths. Speech uses time stretching, added noise and masking of spectrogram bands (SpecAugment); text uses back-translation and paraphrasing by language models. Augmentation encodes prior knowledge about invariances the model should have, reduces overfittingoverfitting, n.: When a model learns its training examples too well — quirks and noise included — so it does great on data it has seen but poorly on anything new, like a student who memorizes answers without understanding. and improves robustness, and it drives contrastive self-supervised learningcontrastive learning, n.: A way to learn without labels by comparison: the model learns to pull together two versions of the same thing, such as two crops of one photo, and push apart unrelated things., where two augmented views of one image form a positive pair. Transformations that are too aggressive can change the true label or create a mismatch with test data. Test-time augmentation averages predictions over augmented copies of an input.
Adding random crops and flips cut the gap between training and validation accuracy in half.
Origin
In statistics, 'data augmentation' first named a different idea: adding latent variables to simplify Bayesian computation (Martin Tanner and Wing Hung Wong, 1987). Training on distorted copies of examples dates at least to 1990s handwriting and digit recognition — Patrice Simard and colleagues' elastic distortions (2003) are a classic — and was central to AlexNet's 2012 ImageNet result.
Usage note
Distinct from generating wholly new synthetic datasynthetic data, n.: Data created by computer programs or AI models rather than collected from the real world — such as practice math problems written by a model, or simulated street scenes for self-driving cars., although the boundary blurs when generative models produce variants of real examples.
Related
- synthetic datasynthetic data, n.: Data created by computer programs or AI models rather than collected from the real world — such as practice math problems written by a model, or simulated street scenes for self-driving cars.
- overfittingoverfitting, n.: When a model learns its training examples too well — quirks and noise included — so it does great on data it has seen but poorly on anything new, like a student who memorizes answers without understanding.
- contrastive learningcontrastive learning, n.: A way to learn without labels by comparison: the model learns to pull together two versions of the same thing, such as two crops of one photo, and push apart unrelated things.
- regularizationregularization, n.: Any technique that stops a model from memorizing its training examples too closely — often by penalizing complexity — so that it works better on new data.
- label smoothinglabel smoothing, n.: A training trick that tells a model the correct answer is very likely rather than absolutely certain, which stops it from becoming overconfident and often helps it generalize.
- equivarianceequivariance, n.: A property of a model whose output changes in step with its input: move the cat in a photo, and the network's internal 'cat' signal moves the same way.
Sources
- Simard, P. Y., Steinkraus, D. & Platt, J. C. (2003). Best Practices for Convolutional Neural Networks Applied to Visual Document Analysis. ICDAR 2003.
- Zhang, H., Cisse, M., Dauphin, Y. N. & Lopez-Paz, D. (2018). mixup: Beyond Empirical Risk Minimization. ICLR 2018.
Mentioned in
few-shot learningfew-shot learning, n.: Getting an AI system to handle a new task or recognize a new kind of thing from just a handful of examples, the way a person can learn a new word after hearing it once or twice., synthetic datasynthetic data, n.: Data created by computer programs or AI models rather than collected from the real world — such as practice math problems written by a model, or simulated street scenes for self-driving cars., AlexNetAlexNet, prop. n.: The neural network that won a major image-recognition contest by a huge margin in 2012, convincing researchers and companies that deep learning trained on graphics chips was the way forward., augmentationaugmentation, n.: Using AI to help people do their work better or faster while they stay in charge, instead of having the AI do the job on its own., data-centric AIdata-centric AI, n.: An approach to building AI that focuses on improving the data — fixing labels, adding missing cases, cleaning inconsistencies — rather than endlessly tweaking the model., dropoutdropout, n.: A training trick that randomly switches off a fraction of a network's neurons at each step, so the network cannot lean too heavily on any one of them and learns sturdier patterns., out-of-distributionout-of-distribution, adj.: Describes inputs that differ from anything an AI system saw in training — like a model trained on daytime photos facing a night scene — where its answers become less reliable., regularizationregularization, n.: Any technique that stops a model from memorizing its training examples too closely — often by penalizing complexity — so that it works better on new data., SMOTESMOTE, abbr.: A technique for lopsided data that invents extra examples of the rare category by blending each rare example with one of its nearest rare neighbors, so a model sees more of it during training., stochasticstochastic, adj.: Involving chance, so the same starting point can lead to different outcomes. AI chatbots are stochastic: ask the same question twice and you may get two different answers., U-NetU-Net, prop. n.: An image network shaped like the letter U: one half shrinks the picture to understand it, the other enlarges it back, with shortcuts between the halves so fine detail is not lost., YOLOYOLO, prop. n.: A family of fast object-detection models that look at an image once and immediately output boxes and labels for everything they recognize, quick enough for live video., Alex KrizhevskyAlex Krizhevsky, prop. n.: A Canadian computer scientist who built AlexNet, the image-recognition network that was trained on gaming graphics chips and whose 2012 contest win launched the deep learning boom., domain randomizationdomain randomization, n.: Training a robot's AI in a simulation whose colors, lighting, weights, friction and other details are shuffled at random on every run, so that the real world holds few surprises it has not already practiced for., invarianceinvariance, n.: A property of a model whose answer does not change when the input is altered in an irrelevant way, such as a photo classifier that still says 'cat' when the cat moves or the picture is flipped., label smoothinglabel smoothing, n.: A training trick that tells a model the correct answer is very likely rather than absolutely certain, which stops it from becoming overconfident and often helps it generalize.