Attention Is All You Need
In plain wordsThe 2017 Google research paper that introduced the transformer, the neural-network design behind ChatGPT, Claude, Gemini and nearly every other modern AI language system.
A paper by eight researchers at Google Brain and Google Research, posted to arXiv in June 2017 and presented at NIPS 2017, that introduced the transformertransformer, n.: The design behind today's chatbots and much of modern AI. It handles text by letting every word look at every other word and decide which of them matter for understanding it. architecture. Its central move was to build a sequence-to-sequence model from attentionattention, n.: A way for an AI model to choose what to focus on. For each word it processes, the model scores how relevant every other word is and draws information mostly from the relevant ones. alone, with no recurrent or convolutional layers: self-attentionself-attention, n.: Attention in which a passage looks at itself: every word checks every other word in the same text to work out its meaning in context, such as which noun 'it' refers to. layers let every position in a sequence draw directly on every other, using scaled dot-product attention split across multiple heads, with positional encodings supplying word order, in an encoder–decoder stack with residual connections and layer normalization. Since attention over a whole sequence is computed at once rather than step by step, the model made far better use of GPUsGPU, abbr.: A computer chip first designed to draw video-game graphics that turned out to be ideal for AI, because it can do thousands of simple calculations at the same time. and TPUsTPU, abbr.: Google's own family of chips built specifically for AI, used to train and run its Gemini models and rented to other companies through its cloud. than recurrent networks did, and it beat the best previous English–German and English–French machine translationmachine translation, n.: Software that renders writing or speech in one language into another language, as online translation services and phone translation apps do. results while costing much less to train. The architecture went on to underlie BERTBERT, prop. n.: An influential 2018 Google language model that learned to understand text by filling in hidden words. It made search engines and many language tools much better at grasping meaning in context., the GPTGPT, abbr.: Short for 'generative pre-trained transformer', the family of OpenAI models behind ChatGPT. The letters are also used loosely for any similar chatbot model. series and virtually all large language modelslarge language model, n.: A computer program trained on enormous amounts of text to predict the words that come next. At vast scale, that simple skill lets it chat, answer questions, translate, summarize and write code., and few twenty-first-century papers have been cited more often.
Nearly every frontier model is still recognizably the architecture of Attention Is All You Need.
Origin
Written by Ashish VaswaniAshish Vaswani, prop. n.: An Indian-born computer scientist listed first on 'Attention Is All You Need', the 2017 paper that introduced the transformer, the design behind today's language models., Noam ShazeerNoam Shazeer, prop. n.: An American computer scientist who co-invented the transformer and several techniques found in nearly every large language model, co-founded Character.AI, and later returned to Google to co-lead Gemini., Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser and Illia Polosukhin, listed as equal contributors in randomized order, and first posted on 12 June 2017 (arXiv:1706.03762). The title plays on the Beatles song 'All You Need Is Love'; Uszkoreit proposed the name Transformer.
Usage note
Attention itself predates the paper: Dzmitry Bahdanau, Kyunghyun Cho and Yoshua Bengio introduced it for neural translation in 2014. All eight authors had left Google by 2023, most to found companies; Shazeer later returned.
Related
- transformertransformer, n.: The design behind today's chatbots and much of modern AI. It handles text by letting every word look at every other word and decide which of them matter for understanding it.
- attentionattention, n.: A way for an AI model to choose what to focus on. For each word it processes, the model scores how relevant every other word is and draws information mostly from the relevant ones.
- self-attentionself-attention, n.: Attention in which a passage looks at itself: every word checks every other word in the same text to work out its meaning in context, such as which noun 'it' refers to.
- large language modellarge language model, n.: A computer program trained on enormous amounts of text to predict the words that come next. At vast scale, that simple skill lets it chat, answer questions, translate, summarize and write code.
- BERTBERT, prop. n.: An influential 2018 Google language model that learned to understand text by filling in hidden words. It made search engines and many language tools much better at grasping meaning in context.
- GPTGPT, abbr.: Short for 'generative pre-trained transformer', the family of OpenAI models behind ChatGPT. The letters are also used loosely for any similar chatbot model.
Sources
- Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, Ł. & Polosukhin, I. (2017). Attention Is All You Need. Advances in Neural Information Processing Systems 30 (NIPS 2017).
Mentioned in
transformertransformer, n.: The design behind today's chatbots and much of modern AI. It handles text by letting every word look at every other word and decide which of them matter for understanding it., arXivarXiv, prop. n.: A free website where scientists post research papers before, or instead of, formal publication. Most new AI research appears there first., CohereCohere, prop. n.: A Canadian AI company, founded in Toronto in 2019, that builds language models and search tools for businesses and governments rather than consumer chatbots., Google BrainGoogle Brain, prop. n.: Google's deep-learning research team from 2011 to 2023, where the transformer — the design behind today's chatbots — and the TensorFlow software were created., machine translationmachine translation, n.: Software that renders writing or speech in one language into another language, as online translation services and phone translation apps do., NeurIPSNeurIPS, prop. n.: The largest yearly scientific conference on artificial intelligence and machine learning, held each December, where many of the field's landmark papers were first presented., Noam ShazeerNoam Shazeer, prop. n.: An American computer scientist who co-invented the transformer and several techniques found in nearly every large language model, co-founded Character.AI, and later returned to Google to co-lead Gemini., Ashish VaswaniAshish Vaswani, prop. n.: An Indian-born computer scientist listed first on 'Attention Is All You Need', the 2017 paper that introduced the transformer, the design behind today's language models., Jakob UszkoreitJakob Uszkoreit, prop. n.: A German computer scientist, one of the eight authors of the transformer paper, credited with first proposing to build language models on attention alone., X is all you needX is all you need, phr.: A joking title formula copied from a famous 2017 AI paper, used to claim that one simple ingredient is enough to solve a problem; hundreds of research papers now use it.