sharpness-aware minimization

An optimization procedure that minimizes the worst-case loss within a small neighborhood of the current parameters — a min–max objective — rather than the loss at a single point, steering training toward flat minimasharp and flat minima, n.: Two kinds of low points a trained model can settle into: narrow pits where small changes spoil performance, and broad basins where they barely matter. Broad basins are thought to generalize better, though this is debated.. Each step computes the gradient, takes an ascent step of radius ρ along its normalized direction to approximate the worst nearby point, computes the gradient there, and applies that second gradient at the original parameters through any base optimizeroptimizer, n.: The rule a training program uses to turn measured errors into actual changes to a model's internal numbers, deciding the direction and size of each adjustment.. The extra gradient computation roughly doubles the cost per step; variants cut the overhead or make the neighborhood scale-invariant (adaptive SAM). SAM improved image-classification benchmarks and robustness to label noiselabel noise, n.: Mistakes in the answers attached to training or test examples — a dog photo marked 'cat' — which can mislead what a model learns and distort how good it seems., with especially large gains for vision transformersvision transformer, n.: A way of applying the transformer, the design behind chatbots, to pictures: the image is cut into small square tiles that the model reads like words in a sentence. and MLP-Mixers trained without heavy augmentation, and is also used in fine-tuning. Why it works is still debated: its benefits do not track measured flatness as closely as its motivation suggests, and its implicit regularization of the training dynamics may matter more.