masked language model
In plain wordsA model trained by hiding some words in a sentence and learning to guess them from the words on both sides, like a fill-in-the-blanks exercise. BERT is the famous example.
A language modellanguage model, n.: A system that has learned how likely different sequences of words are, so it can predict the next word in a text and, by doing that repeatedly, write text of its own. trained to recover tokens that have been hidden or corrupted in its input, using context on both sides. In BERTBERT, prop. n.: An influential 2018 Google language model that learned to understand text by filling in hidden words. It made search engines and many language tools much better at grasping meaning in context.'s recipe, 15% of input tokens are selected; of these, 80% are replaced by a [MASK] symbol, 10% by a random token and 10% left unchanged, and the model is trained with cross-entropycross-entropy, n.: A score of how surprised a model's predictions are by what actually happens; lower is better. Language models are trained mainly by driving this number down on real text. to predict the originals. Because attention is bidirectional (no causal maskcausal mask, n.: A rule inside text-generating AI that stops each word from seeing the words after it, so the model learns to predict what comes next without being able to cheat.), every token's representation draws on the whole input, which suits classification, retrieval, tagging and extraction after fine-tuningfine-tuning, n.: Taking an AI model that has already learned a lot and training it a little more on a smaller, focused set of examples so it gets better at one job or behaves a certain way.. An MLM defines no simple left-to-right distribution over text, so it is poor at open-ended generation, and only masked positions provide training signal. Variants mask whole spans (SpanBERT, T5T5, prop. n.: A 2019 Google model that treats every language task — translating, summarizing, answering, classifying — as turning one piece of text into another, using one shared design.'s span corruption) or detect replaced tokens (ELECTRA); masked diffusion language modelsdiffusion language model, n.: A language model that writes by starting from a fully blanked-out answer and filling it in over several rounds, settling many words at once instead of writing strictly left to right. vary the masking rate to make generation possible.
We continued MLM pretraining on clinical notes before fine-tuning the classifier.
Origin
Introduced as the 'masked LM' pretraining task of BERT by Jacob Devlin, Ming-Wei Chang, Kenton Lee and Kristina Toutanova of Google (October 2018), who related it to the cloze procedure proposed by the communication researcher Wilson L. Taylor in 1953 for measuring readability.
Related
- BERTBERT, prop. n.: An influential 2018 Google language model that learned to understand text by filling in hidden words. It made search engines and many language tools much better at grasping meaning in context.
- encoder-only transformerencoder-only transformer, n.: A transformer that reads a whole text at once, in both directions, to understand it rather than to write new text. Such models power search, sorting documents by topic and spotting names in text.
- causal language modelcausal language model, n.: A language model that reads text only from left to right, predicting each word from the words before it. The models behind today's chatbots are of this kind.
- self-supervised learningself-supervised learning, n.: A way for AI to learn from huge amounts of unlabeled data by setting itself fill-in-the-blank puzzles, such as guessing a hidden word or a missing patch of an image.
- diffusion language modeldiffusion language model, n.: A language model that writes by starting from a fully blanked-out answer and filling it in over several rounds, settling many words at once instead of writing strictly left to right.
- attention maskattention mask, n.: A grid of yes-or-no switches telling an attention layer which words each word may look at — for example, ignore blank padding, or do not peek at words that come later.
- causal language modelcausal language model, n.: A language model that reads text only from left to right, predicting each word from the words before it. The models behind today's chatbots are of this kind.
Sources
- Devlin, J., Chang, M.-W., Lee, K. & Toutanova, K. (2019). BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. NAACL 2019. arxiv.org/abs/1810.04805
- Taylor, W. L. (1953). 'Cloze procedure': A new tool for measuring readability. Journalism Quarterly 30(4).
Mentioned in
language modellanguage model, n.: A system that has learned how likely different sequences of words are, so it can predict the next word in a text and, by doing that repeatedly, write text of its own., next-token predictionnext-token prediction, n.: The simple task most AI chatbots were trained on: given some text, guess the next small piece of it. Repeating that guess, piece by piece, produces whole answers., pretrainingpretraining, n.: The first and usually the most expensive stage of building a modern AI model, in which it learns general patterns from enormous amounts of raw text, images or code before being specialized for particular uses., attention maskattention mask, n.: A grid of yes-or-no switches telling an attention layer which words each word may look at — for example, ignore blank padding, or do not peek at words that come later., autoregressive modelautoregressive model, n.: A model that produces a sequence one piece at a time, each new piece depending on the pieces already produced — like writing a sentence word by word without going back., BERTBERT, prop. n.: An influential 2018 Google language model that learned to understand text by filling in hidden words. It made search engines and many language tools much better at grasping meaning in context., causal language modelcausal language model, n.: A language model that reads text only from left to right, predicting each word from the words before it. The models behind today's chatbots are of this kind., causal maskcausal mask, n.: A rule inside text-generating AI that stops each word from seeing the words after it, so the model learns to predict what comes next without being able to cheat., encoder-only transformerencoder-only transformer, n.: A transformer that reads a whole text at once, in both directions, to understand it rather than to write new text. Such models power search, sorting documents by topic and spotting names in text., self-supervised learningself-supervised learning, n.: A way for AI to learn from huge amounts of unlabeled data by setting itself fill-in-the-blank puzzles, such as guessing a hidden word or a missing patch of an image., denoising autoencoderdenoising autoencoder, n.: An autoencoder trained to recover clean data from a deliberately damaged copy, such as an image with random pixels erased, which teaches it the structure of the data rather than how to copy., diffusion language modeldiffusion language model, n.: A language model that writes by starting from a fully blanked-out answer and filling it in over several rounds, settling many words at once instead of writing strictly left to right., fill-in-the-middlefill-in-the-middle, n.: A training trick that teaches a text-writing AI to fill a gap using what comes both before and after it — essential for coding assistants that insert code in the middle of a file., masked autoencodermasked autoencoder, n.: A way to teach a vision model without labels: hide most of an image's tiles and train the model to fill in the missing parts, so that it learns what images are made of.