Skip to content
Last Dictionary
Plane
Language Models & TransformersCore ConceptsMathematics & Statistics

language model

nounAlso: LM, language modeling, language modelling, statistical language model, neural language model

In plain wordsA system that has learned how likely different sequences of words are, so it can predict the next word in a text and, by doing that repeatedly, write text of its own.

  1. A model that estimates how likely any piece of text is, usually by learning, at every position, the probability of each possible next tokentoken, n.: The small chunk of text an AI language model reads and writes — often a word, part of a word or a punctuation mark. AI usage, limits and prices are counted in tokens. (word, subword, character or byte) given the tokens before it; multiplying these conditional probabilities by the chain rule gives the probability of a whole sequence. Such a model can therefore score text (summarized by perplexityperplexity, n.: A score for how surprised a language model is by real text; lower is better. A perplexity of 10 means guessing each next word is, on average, as hard for the model as guessing a ten-sided die's roll.) and generate it by decodingdecoding, n.: The method a language model uses to turn its predictions into actual text — for example, always picking the most likely next word, or choosing somewhat at random among the likely ones.. Language models have been built from n-gram countsn-gram model, n.: An early type of language model that predicts the next word from just the few words before it, using counts of how often each word followed those words in a large collection of text., recurrent networksrecurrent neural network, n.: A neural network that reads a sequence, such as a sentence or an audio clip, one piece at a time, carrying a running summary forward so that what came earlier can shape how later pieces are understood. and, since 2018, transformerstransformer, n.: The design behind today's chatbots and much of modern AI. It handles text by letting every word look at every other word and decide which of them matter for understanding it.. The term also covers models trained with other text objectives, such as masked language modelsmasked language model, n.: A model trained by hiding some words in a sentence and learning to guess them from the words on both sides, like a fill-in-the-blanks exercise. BERT is the famous example. that fill in blanks rather than continue text, and diffusion language modelsdiffusion language model, n.: A language model that writes by starting from a fully blanked-out answer and filling it in over several rounds, settling many words at once instead of writing strictly left to right.. Trained on very large corpora, language models absorb much of the knowledge and skill expressed in text, which is what makes large language modelslarge language model, n.: A computer program trained on enormous amounts of text to predict the words that come next. At vast scale, that simple skill lets it chat, answer questions, translate, summarize and write code. useful as general-purpose systems.

    We use a small language model to filter low-quality web pages before pretraining.

p(x1,…,xT)=∏t=1Tp(xt∣x1,…,xt−1)p(x_1, \dots, x_T) = \prod_{t=1}^{T} p(x_t \mid x_1, \dots, x_{t-1})

Origin

Statistical modeling of text began with Andrey Markov's 1913 letter-sequence study and Claude Shannon's 1948 and 1951 work on predicting English. The phrase 'language model' became established in statistical speech recognition, notably in Frederick Jelinek's group at IBM from the 1970s. The neural probabilistic language model of Yoshua Bengio and colleagues (2000–2003) began the shift to neural networks.

Usage note

In everyday speech 'language model' often means a large language modellarge language model, n.: A computer program trained on enormous amounts of text to predict the words that come next. At vast scale, that simple skill lets it chat, answer questions, translate, summarize and write code. or a chatbot built on one; technically, any model assigning probabilities to text qualifies, however small.

Related

See also
  • large language modellarge language model, n.: A computer program trained on enormous amounts of text to predict the words that come next. At vast scale, that simple skill lets it chat, answer questions, translate, summarize and write code.
  • n-gram modeln-gram model, n.: An early type of language model that predicts the next word from just the few words before it, using counts of how often each word followed those words in a large collection of text.
  • next-token predictionnext-token prediction, n.: The simple task most AI chatbots were trained on: given some text, guess the next small piece of it. Repeating that guess, piece by piece, produces whole answers.
  • autoregressive modelautoregressive model, n.: A model that produces a sequence one piece at a time, each new piece depending on the pieces already produced — like writing a sentence word by word without going back.
  • masked language modelmasked language model, n.: A model trained by hiding some words in a sentence and learning to guess them from the words on both sides, like a fill-in-the-blanks exercise. BERT is the famous example.
  • perplexityperplexity, n.: A score for how surprised a language model is by real text; lower is better. A perplexity of 10 means guessing each next word is, on average, as hard for the model as guessing a ten-sided die's roll.
  • transformertransformer, n.: The design behind today's chatbots and much of modern AI. It handles text by letting every word look at every other word and decide which of them matter for understanding it.

Sources

  1. Shannon, C. E. (1951). Prediction and Entropy of Printed English. Bell System Technical Journal 30.
  2. Bengio, Y., Ducharme, R., Vincent, P. & Jauvin, C. (2003). A Neural Probabilistic Language Model. Journal of Machine Learning Research 3.

Mentioned in

chain of thoughtchain of thought, n.: Getting an AI to write out its reasoning step by step before giving an answer, the way a student shows their working. This often makes answers to math and logic problems more accurate., context windowcontext window, n.: How much an AI model can read and keep in view at once: the instructions, the conversation so far and any attached documents. Text that does not fit has to be cut, summarized or left out., GPTGPT, abbr.: Short for 'generative pre-trained transformer', the family of OpenAI models behind ChatGPT. The letters are also used loosely for any similar chatbot model., large language modellarge language model, n.: A computer program trained on enormous amounts of text to predict the words that come next. At vast scale, that simple skill lets it chat, answer questions, translate, summarize and write code., modelmodel, n.: A simplified working version of something. In AI it usually means the trained program, a big set of learned numbers, that takes an input and produces an answer, prediction or piece of content., natural language processingnatural language processing, n.: The field that teaches computers to handle human language — to read, interpret, translate, summarize and produce text and speech. It is one of the oldest and largest areas of artificial intelligence., next-token predictionnext-token prediction, n.: The simple task most AI chatbots were trained on: given some text, guess the next small piece of it. Repeating that guess, piece by piece, produces whole answers., optical character recognitionoptical character recognition, n.: Software that picks out the letters and words in a picture — a road sign, a handwritten note, a page of an old book — and turns them into real text a computer can copy, search or read aloud., promptprompt, n.: The message or instructions you give an AI system — a question, a request, some examples or a document — that it responds to. Clearer prompts generally get better results., speech recognitionspeech recognition, n.: Technology that turns spoken words into written text, as when a phone transcribes a voice message or captions appear automatically under a video., tokentoken, n.: The small chunk of text an AI language model reads and writes — often a word, part of a word or a punctuation mark. AI usage, limits and prices are counted in tokens., audio language modelaudio language model, n.: An AI language model that can take in sound, such as speech, music and everyday noises, and understand or generate it, much as text chatbots handle words., base modelbase model, n.: A language model fresh from its first broad training on text, before being taught to follow instructions or chat. It continues whatever text it is given instead of answering., causal language modelcausal language model, n.: A language model that reads text only from left to right, predicting each word from the words before it. The models behind today's chatbots are of this kind., chat modelchat model, n.: A language model trained to hold a back-and-forth conversation, keeping track of who said what and replying as a helpful assistant. It is the engine inside chat apps., completioncompletion, n.: The text an AI language model writes in response to a prompt — literally, its continuation of the text it was given., constrained decodingconstrained decoding, n.: Forcing an AI model's output to follow a required format — such as valid code or a fill-in form — by blocking, at every step, any word that would break the rules., Emily M. BenderEmily M. Bender, prop. n.: An American linguist who argues that language models only mimic the form of language without understanding it, and who co-coined the phrase 'stochastic parrots'., fancy autocompletefancy autocomplete, n.: A dismissive description of AI chatbots as nothing more than a souped-up version of the feature on a phone keyboard that suggests your next word., GPT-2GPT-2, prop. n.: A 2019 OpenAI language model that wrote surprisingly fluent paragraphs. OpenAI at first withheld the full version, saying it could be misused for fake news, which made it famous., GPT-3GPT-3, prop. n.: A huge 2020 OpenAI language model that could perform new tasks simply from instructions or a few examples written into its prompt, without retraining. It made prompting a new way of programming., I. J. GoodI. J. Good, prop. n.: A British statistician who broke codes with Alan Turing and in 1965 argued that a machine smarter than us could design still smarter machines, setting off an 'intelligence explosion'., instruction-tuned modelinstruction-tuned model, n.: A language model given extra training on examples of instructions paired with good responses, so it does what you ask instead of merely continuing your text., Jared KaplanJared Kaplan, prop. n.: An American physicist who showed that language models improve predictably as they grow and train on more data, and who co-founded the AI company Anthropic.