scalar

A single element of the number field over which vectors are defined — in machine learning almost always a real number stored in floating point. Multiplying vectorsvector, n.: An ordered list of numbers, like map coordinates. AI systems turn words, images and users into long lists of numbers so that similar things end up as nearby points. and matricesmatrix, n.: A rectangular grid of numbers arranged in rows and columns. Most of what a neural network has learned is stored in large grids like this. by a scalar scales every entry, and scalars can be treated as zero-axis tensorstensor, n.: A block of numbers with any number of dimensions: a single number, a list, a grid, or a stack of grids. AI software stores and processes all its data this way.. Training funnels everything through scalars at key points: the lossloss function, n.: A score that measures how wrong a model's predictions are. Training means adjusting the model, step by step, to push this number down. that backpropagationbackpropagation, n.: The method neural networks use to learn from mistakes: it works backward from the error to figure out how much each internal connection contributed, so every connection can be adjusted in the right direction. differentiates must be a single scalar, and the learning ratelearning rate, n.: How big a step a model takes each time it adjusts its settings during training. Too large and learning goes haywire; too small and it crawls., temperaturetemperature, n.: A setting that controls how adventurous an AI model's word choices are. Low temperature makes it pick the safest, most likely words; high temperature makes it more varied and surprising, but more error-prone. and regularization strength are scalar hyperparametershyperparameter, n.: A setting chosen by people before training, such as how fast a model learns or how big it is, as opposed to the internal values the model learns by itself from data.. In array libraries a scalar may be a zero-dimensional array, a distinction that matters for broadcasting rules and for when values are copied between GPUGPU, abbr.: A computer chip first designed to draw video-game graphics that turned out to be ideal for AI, because it can do thousands of simple calculations at the same time. and CPU memory.

Call backward() only on a scalar loss, so average the per-token losses first.