logit
In plain wordsA raw score that a model gives each possible answer before the scores are turned into probabilities. In statistics, the 'log-odds': a probability rewritten on a scale running from minus to plus infinity.
- 1
Neural networks
The unnormalized score that a neural networkneural network, n.: A kind of computer program that learns a task from examples by tuning thousands to billions of connections between simple number-crunching units, an arrangement loosely inspired by nerve cells in the brain.'s final layer assigns to each class or tokentoken, n.: The small chunk of text an AI language model reads and writes — often a word, part of a word or a punctuation mark. AI usage, limits and prices are counted in tokens. before softmaxsoftmax, n.: A formula that turns a list of raw scores into probabilities — all positive and adding up to one — with the biggest share going to the highest score. (or, for a binary output, the sigmoidsigmoid function, n.: An S-shaped curve that squeezes any number into a value between 0 and 1, so the result can be read as a probability or as how far a switch is turned on.) converts the scores into probabilities. Logits are real numbers of any sign; only their differences matter to softmax, so adding the same constant to all of them changes nothing. A language modellarge language model, n.: A computer program trained on enormous amounts of text to predict the words that come next. At vast scale, that simple skill lets it chat, answer questions, translate, summarize and write code. outputs one logit per vocabulary entry at each position, and decoding controls act on them: temperaturetemperature, n.: A setting that controls how adventurous an AI model's word choices are. Low temperature makes it pick the safest, most likely words; high temperature makes it more varied and surprising, but more error-prone. divides them, logit biaslogit bias, n.: A setting that lets developers make specific words more or less likely in an AI model's output — even ban a word outright — by adjusting the model's scores before it chooses. adds to them, and top-k and top-p sampling truncate the resulting distribution. Training code passes logits rather than probabilities to the loss because a fused softmax and cross-entropycross-entropy, n.: A score of how surprised a model's predictions are by what actually happens; lower is better. Language models are trained mainly by driving this number down on real text. computed from logits is numerically stable. In interpretabilityinterpretability, n.: The effort to understand how an AI system arrives at its outputs — what information it uses and how it combines it — instead of treating it as a sealed black box., the logit lenslogit lens, n.: A quick trick for peeking at what a language model is 'thinking' partway through: read its half-finished internal state as if it were the final answer, one layer at a time. reads intermediate layers in logit space.
Pass the raw logits to the loss function; don't apply softmax first.
- 2
Statistics
The log-odds of a probability p, logit(p) = ln(p/(1 − p)), which maps the interval (0, 1) onto the whole real line and is the inverse of the logistic sigmoid functionsigmoid function, n.: An S-shaped curve that squeezes any number into a value between 0 and 1, so the result can be read as a probability or as how far a switch is turned on.. In logistic regressionlogistic regression, n.: A simple, widely used method for yes-or-no predictions that outputs a probability — say, a 70% chance a customer cancels — by passing a weighted sum of inputs through an S-shaped curve. the logit of the outcome probability is modeled as a linear function of the predictors, so each coefficient is a change in log-odds per unit change of its predictor. The logit is the canonical link function for binomial data in generalized linear modelsgeneralized linear model, n.: A family of statistical models that extends straight-line regression to outcomes such as yes/no answers, counts or waiting times, by linking the average outcome to a weighted sum of inputs through a suitable curve..
Origin
Coined by statistician Joseph Berkson in 'Application of the Logistic Function to Bio-Assay' (Journal of the American Statistical Association, 1944), short for 'logistic unit', by analogy with Chester Bliss's 'probit' (1934). The looser neural-network sense, any pre-softmax score, spread with deep-learning practice and software in the 2010s; TensorFlow (2015), for example, names the inputs of its fused softmax–cross-entropy losses 'logits'.
Usage note
The machine-learning sense stretches the statistical one: a sigmoid classifier's logit really is a log-odds, but softmax logits equal log-probabilities only up to an additive constant. Practitioners also speak of attention logits, the scores before attention's softmax.
Related
- softmaxsoftmax, n.: A formula that turns a list of raw scores into probabilities — all positive and adding up to one — with the biggest share going to the highest score.
- sigmoid functionsigmoid function, n.: An S-shaped curve that squeezes any number into a value between 0 and 1, so the result can be read as a probability or as how far a switch is turned on.
- logistic regressionlogistic regression, n.: A simple, widely used method for yes-or-no predictions that outputs a probability — say, a 70% chance a customer cancels — by passing a weighted sum of inputs through an S-shaped curve.
- log probabilitylog probability, n.: The logarithm of the probability a model gives to a token or a text. Computers use it because multiplying many tiny probabilities runs into rounding problems, whereas adding their logarithms does not.
- temperaturetemperature, n.: A setting that controls how adventurous an AI model's word choices are. Low temperature makes it pick the safest, most likely words; high temperature makes it more varied and surprising, but more error-prone.
- logit lenslogit lens, n.: A quick trick for peeking at what a language model is 'thinking' partway through: read its half-finished internal state as if it were the final answer, one layer at a time.
- cross-entropycross-entropy, n.: A score of how surprised a model's predictions are by what actually happens; lower is better. Language models are trained mainly by driving this number down on real text.
- log probabilitylog probability, n.: The logarithm of the probability a model gives to a token or a text. Computers use it because multiplying many tiny probabilities runs into rounding problems, whereas adding their logarithms does not.
Sources
- Berkson, J. (1944). Application of the Logistic Function to Bio-Assay. Journal of the American Statistical Association, 39(227), 357–365.
Mentioned in
temperaturetemperature, n.: A setting that controls how adventurous an AI model's word choices are. Low temperature makes it pick the safest, most likely words; high temperature makes it more varied and surprising, but more error-prone., calibrationcalibration, n.: How well a model's stated confidence matches reality: when a well-calibrated system says it is 80% sure, it turns out to be right about 80% of the time., categorical distributioncategorical distribution, n.: The probabilities for a single pick among a fixed set of options, like a weighted die. A language model's guess about the next word is one of these, spread over its whole vocabulary., constrained decodingconstrained decoding, n.: Forcing an AI model's output to follow a required format — such as valid code or a fill-in form — by blocking, at every step, any word that would break the rules., decodingdecoding, n.: The method a language model uses to turn its predictions into actual text — for example, always picking the most likely next word, or choosing somewhat at random among the likely ones., greedy decodinggreedy decoding, n.: The simplest way for a language model to write: always pick the single most likely next word. It is fast and predictable but can produce dull or repetitive text., log probabilitylog probability, n.: The logarithm of the probability a model gives to a token or a text. Computers use it because multiplying many tiny probabilities runs into rounding problems, whereas adding their logarithms does not., logistic regressionlogistic regression, n.: A simple, widely used method for yes-or-no predictions that outputs a probability — say, a 70% chance a customer cancels — by passing a weighted sum of inputs through an S-shaped curve., model extractionmodel extraction, n.: Copying an AI model without permission by asking it huge numbers of questions and using the answers to build a look-alike, or to work out hidden details of how it was made., numerical stabilitynumerical stability, n.: Whether a computer calculation stays accurate despite the tiny rounding errors in every step, instead of blowing up into nonsense such as infinities., prefillprefill, n.: The first stage of an AI's reply, in which it reads your whole prompt in one go before it starts writing. Long prompts make this stage take longer., repetition penaltyrepetition penalty, n.: A setting that discourages an AI model from repeating itself by lowering the chances of words it has already used, so its text does not get stuck in loops., residual streamresidual stream, n.: The running internal state that flows through a transformer model from layer to layer; each layer reads from it and adds its own contribution, like editors annotating a shared document., sigmoid functionsigmoid function, n.: An S-shaped curve that squeezes any number into a value between 0 and 1, so the result can be read as a probability or as how far a switch is turned on., softmaxsoftmax, n.: A formula that turns a list of raw scores into probabilities — all positive and adding up to one — with the biggest share going to the highest score., tanhtanh, n.: An S-shaped curve, like the sigmoid but centered on zero, that squeezes any number into a value between −1 and 1. It was the standard neuron rule before ReLU took over., vocabularyvocabulary, n.: The fixed list of all the tokens a language model knows, each with its own number. Everything the model reads or writes must be built from pieces on this list., attribution graphattribution graph, n.: A diagram showing which internal concepts led an AI model to a particular answer, with arrows showing how strongly each concept pushed the next., Carlini–Wagner attackCarlini–Wagner attack, n.: A careful, optimization-based way of finding the smallest change to an input that fools a neural network; it broke several defenses that had looked secure., circuit tracingcircuit tracing, n.: Methods that map, step by step, which internal concepts an AI model used to produce a particular answer and how they fed into one another, like tracing wires on a circuit board., decision modeldecision model, n.: A kind of AI language model that reads text but answers only with numbers — a yes-or-no probability, a choice or a score — instead of writing words., direct logit attributiondirect logit attribution, n.: A quick way to measure how much each internal part of a language model directly pushed it toward a particular next word, by splitting the final score into separate contributions., label smoothinglabel smoothing, n.: A training trick that tells a model the correct answer is very likely rather than absolutely certain, which stops it from becoming overconfident and often helps it generalize., logit biaslogit bias, n.: A setting that lets developers make specific words more or less likely in an AI model's output — even ban a word outright — by adjusting the model's scores before it chooses.