Adafactor

An adaptive optimizeroptimizer, n.: The rule a training program uses to turn measured errors into actual changes to a model's internal numbers, deciding the direction and size of each adjustment. designed to cut the memory cost of AdamAdam, prop. n.: The most widely used method for training neural networks. It smooths the direction of updates over time and automatically sets a separate step size for each setting based on how large its recent changes have been.-style second-moment statistics. Instead of a full matrix of second-moment estimates for each weight matrix, it stores one vector of row statistics and one of column statistics and rebuilds an approximate per-parameter value from their outer product (a rank-one factorization), cutting the state for an n × m matrix from nm numbers to n + m. By default it also drops first-moment momentummomentum, n.: A training trick that lets each update keep part of the previous update's direction, like a ball rolling downhill that builds up speed and coasts over small bumps., clips updates whose root-mean-square exceeds a threshold, raises the second-moment decay rate over training, and can scale step sizes relative to each parameter's magnitude. These choices made it the optimizer of choice for memory-constrained transformertransformer, n.: The design behind today's chatbots and much of modern AI. It handles text by letting every word look at every other word and decide which of them matter for understanding it. training on TPUs, including T5 (2019) and, in modified form, PaLM (2022). Its row-and-column statistics were later linked to ShampooShampoo, prop. n.: A training method that adjusts each weight table's updates using statistics gathered along its rows and its columns, capturing how weights interact. Each step costs more computing, but fewer steps are needed. by the SOAP analysis, and similar factorizations recur in later memory-efficient optimizers.