Skip to content
Last Dictionary
Plane
Language Models & TransformersMathematics & StatisticsTraining & Post-training

temperature

nounAlso: sampling temperature, softmax temperature, temperature scaling, temp

In plain wordsA setting that controls how adventurous an AI model's word choices are. Low temperature makes it pick the safest, most likely words; high temperature makes it more varied and surprising, but more error-prone.

  1. 1

    Sampling

    A positive number T by which logitslogit, n.: A raw score that a model gives each possible answer before the scores are turned into probabilities. In statistics, the 'log-odds': a probability rewritten on a scale running from minus to plus infinity. are divided before the softmaxsoftmax, n.: A formula that turns a list of raw scores into probabilities — all positive and adding up to one — with the biggest share going to the highest score. when sampling from a model, controlling how spread out the output distribution is: T < 1 sharpens it toward the most likely tokenstoken, n.: The small chunk of text an AI language model reads and writes — often a word, part of a word or a punctuation mark. AI usage, limits and prices are counted in tokens., T > 1 flattens it toward uniform, T = 1 leaves the model's distribution unchanged, and T approaching 0 approaches greedy decodinggreedy decoding, n.: The simplest way for a language model to write: always pick the single most likely next word. It is fast and predictable but can produce dull or repetitive text.. Offered in nearly all LLM APIs, commonly within 0–1 or 0–2, it is the main dial between consistency and diversity and is often combined with truncation such as top-p samplingtop-p sampling, n.: A way of making AI-written text varied but sensible: the model picks at random among the smallest group of likely next words whose chances add up to a set share, such as 90%.. Higher values increase variety for brainstorming or best-of-n samplingbest-of-n sampling, n.: Asking an AI the same question several times and keeping whichever answer a scoring model or checker rates highest — a simple way to trade extra computing for better answers. but raise error and incoherence rates; temperature 0 is common for extraction and evaluation, though it does not guarantee identical outputs (nondeterminismnondeterminism, n.: The fact that an AI model can give different answers to exactly the same question, partly because it deliberately chooses words with some randomness and partly because of tiny computing differences.).

    Set temperature to 0.2 for the data extraction and 0.9 for the slogan ideas.

  2. 2

    Training and calibration

    The same logit-scaling parameter used for other purposes: in knowledge distillationknowledge distillation, n.: Training a smaller or cheaper 'student' AI to copy the behavior of a larger 'teacher' AI, so the student keeps much of the teacher's skill while being faster to run., a high T softens teacher and student distributions so that small probabilities carry information; in calibrationcalibration, n.: How well a model's stated confidence matches reality: when a well-calibrated system says it is 80% sure, it turns out to be right about 80% of the time., temperature scaling fits a single T on held-out data so a classifier's confidence matches its accuracy; in contrastive learning, a temperature hyperparameter sharpens similarity scores.

  3. 3

    Physics and optimization

    The parameter of the Boltzmann distribution from statistical mechanics, from which the machine-learning usage is borrowed; in simulated annealingsimulated annealing, n.: An optimization method inspired by slowly cooling metal: early on it sometimes accepts worse solutions, to escape dead ends, and becomes pickier as a 'temperature' setting gradually drops. and Boltzmann machinesBoltzmann machine, n.: A network of randomly flipping on–off units, including hidden ones, that learns the statistical patterns in its training data and can then produce new examples like them., the temperature is gradually lowered so that a stochastic search settles into low-energy states.

pi=exp⁡(zi/T)∑jexp⁡(zj/T)p_i = \frac{\exp(z_i / T)}{\sum_j \exp(z_j / T)}

Origin

Borrowed from statistical mechanics via the Boltzmann distribution. Simulated annealing (Scott Kirkpatrick, C. Daniel Gelatt and Mario Vecchi, 1983) and Boltzmann machines (David Ackley, Geoffrey Hinton and Terrence Sejnowski, 1985) carried it into computing; Hinton, Oriol Vinyals and Jeff Dean's distillation paper (2015) and text-generation APIs from 2020 made it everyday vocabulary.

Usage note

Temperature does not make a model more creative in any deep sense; it only reshapes probabilities the model already assigns. Some reasoning models fix or restrict the temperature setting.

Related

See also
  • softmaxsoftmax, n.: A formula that turns a list of raw scores into probabilities — all positive and adding up to one — with the biggest share going to the highest score.
  • top-p samplingtop-p sampling, n.: A way of making AI-written text varied but sensible: the model picks at random among the smallest group of likely next words whose chances add up to a set share, such as 90%.
  • top-k samplingtop-k sampling, n.: A way of making AI-written text varied but sensible: at each step the model picks at random, but only among its k most likely next words, such as the top 40.
  • greedy decodinggreedy decoding, n.: The simplest way for a language model to write: always pick the single most likely next word. It is fast and predictable but can produce dull or repetitive text.
  • decodingdecoding, n.: The method a language model uses to turn its predictions into actual text — for example, always picking the most likely next word, or choosing somewhat at random among the likely ones.
  • knowledge distillationknowledge distillation, n.: Training a smaller or cheaper 'student' AI to copy the behavior of a larger 'teacher' AI, so the student keeps much of the teacher's skill while being faster to run.
  • simulated annealingsimulated annealing, n.: An optimization method inspired by slowly cooling metal: early on it sometimes accepts worse solutions, to escape dead ends, and becomes pickier as a 'temperature' setting gradually drops.

Sources

  1. Hinton, G., Vinyals, O. & Dean, J. (2015). Distilling the Knowledge in a Neural Network. NIPS 2014 Deep Learning Workshop; arXiv 2015.
  2. Ackley, D. H., Hinton, G. E. & Sejnowski, T. J. (1985). A Learning Algorithm for Boltzmann Machines. Cognitive Science 9(1).

Mentioned in

knowledge distillationknowledge distillation, n.: Training a smaller or cheaper 'student' AI to copy the behavior of a larger 'teacher' AI, so the student keeps much of the teacher's skill while being faster to run., model APImodel API, n.: A web service through which programs send requests to an AI model run by a company and receive its answers, usually paying per use instead of running the model themselves., calibrationcalibration, n.: How well a model's stated confidence matches reality: when a well-calibrated system says it is 80% sure, it turns out to be right about 80% of the time., categorical distributioncategorical distribution, n.: The probabilities for a single pick among a fixed set of options, like a weighted die. A language model's guess about the next word is one of these, spread over its whole vocabulary., classifier-free guidanceclassifier-free guidance, n.: A setting that makes an image generator follow the prompt more strictly by comparing what it would draw with and without the prompt, then exaggerating the difference., CLIPCLIP, prop. n.: An OpenAI model that learned to match pictures with their captions from hundreds of millions of web examples, so it can judge how well any piece of text describes any image., contrastive learningcontrastive learning, n.: A way to learn without labels by comparison: the model learns to pull together two versions of the same thing, such as two crops of one photo, and push apart unrelated things., decodingdecoding, n.: The method a language model uses to turn its predictions into actual text — for example, always picking the most likely next word, or choosing somewhat at random among the likely ones., greedy decodinggreedy decoding, n.: The simplest way for a language model to write: always pick the single most likely next word. It is fast and predictable but can produce dull or repetitive text., hyperparameterhyperparameter, n.: A setting chosen by people before training, such as how fast a model learns or how big it is, as opposed to the internal values the model learns by itself from data., logitlogit, n.: A raw score that a model gives each possible answer before the scores are turned into probabilities. In statistics, the 'log-odds': a probability rewritten on a scale running from minus to plus infinity., mode collapsemode collapse, n.: When a generator gets stuck producing only a few kinds of output — the same face, the same joke — instead of the full variety it was meant to learn., nondeterminismnondeterminism, n.: The fact that an AI model can give different answers to exactly the same question, partly because it deliberately chooses words with some randomness and partly because of tiny computing differences., OpenAI-compatible APIOpenAI-compatible API, n.: An AI service that accepts the same request format as OpenAI's, so software written for OpenAI's models can use other companies' or self-hosted models by changing an address and key., parameterparameter, n.: One of the adjustable numbers inside an AI model that are tuned during training. Large language models have billions of them, and together they store what the model has learned., random seedrandom seed, n.: A number that fixes the 'random' choices a computer makes; reuse the same seed and settings, and an image generator produces the same picture again., repetition penaltyrepetition penalty, n.: A setting that discourages an AI model from repeating itself by lowering the chances of words it has already used, so its text does not get stuck in loops., samplingsampling, n.: Picking some items at random from a larger set or from a range of possibilities — surveying 1,000 voters, say, or letting an AI choose its next word by a weighted lottery., scalarscalar, n.: A single number, such as 3.7 or −2, as opposed to a list or grid of numbers. A model's error score and its learning rate are both scalars., self-consistencyself-consistency, n.: Asking an AI the same question many times, letting it reason differently each time, then going with the answer it reaches most often — a vote among its own attempts., simulated annealingsimulated annealing, n.: An optimization method inspired by slowly cooling metal: early on it sometimes accepts worse solutions, to escape dead ends, and becomes pickier as a 'temperature' setting gradually drops., softmaxsoftmax, n.: A formula that turns a list of raw scores into probabilities — all positive and adding up to one — with the biggest share going to the highest score., top-k samplingtop-k sampling, n.: A way of making AI-written text varied but sensible: at each step the model picks at random, but only among its k most likely next words, such as the top 40., top-p samplingtop-p sampling, n.: A way of making AI-written text varied but sensible: the model picks at random among the smallest group of likely next words whose chances add up to a set share, such as 90%.