positional encoding
In plain wordsExtra information added to each word so a transformer knows where it sits in the text. Without it, 'dog bites man' and 'man bites dog' would look the same to the model.
Any method of giving a transformertransformer, n.: The design behind today's chatbots and much of modern AI. It handles text by letting every word look at every other word and decide which of them matter for understanding it. information about token order, which self-attentionself-attention, n.: Attention in which a passage looks at itself: every word checks every other word in the same text to work out its meaning in context, such as which noun 'it' refers to. alone ignores because it treats its input as an unordered set. Absolute schemes add to each tokentoken, n.: The small chunk of text an AI language model reads and writes — often a word, part of a word or a punctuation mark. AI usage, limits and prices are counted in tokens. embeddingembedding, n.: A list of numbers that stands for a word, picture or other item, arranged so that similar items get similar lists. It lets a computer measure similarity of meaning as distance. a vector that depends on its index: the original transformer used fixed sinusoids with geometrically spaced frequencies, while BERTBERT, prop. n.: An influential 2018 Google language model that learned to understand text by filling in hidden words. It made search engines and many language tools much better at grasping meaning in context. and GPT-2 learned a table of position embeddings up to a maximum length. Relative schemes make attention depend on the distance between query and key, through learned relative embeddings, learned per-distance biases (as in T5T5, prop. n.: A 2019 Google model that treats every language task — translating, summarizing, answering, classifying — as turning one piece of text into another, using one shared design.), linear distance penalties (ALiBiALiBi, abbr.: A simple way to give a transformer a sense of word order: attention between two words is penalized more the farther apart they are, which helps it handle texts longer than those it trained on.) or rotations (rotary positional embeddingrotary positional embedding, n.: A way of telling a transformer where each word is by rotating parts of its internal numbers by an angle that grows with position, so the model can read off how far apart two words are., now the most common choice). The scheme largely determines how well a model copes with sequences longer than those seen in training. Some decoder-only models use no explicit encoding at all (NoPE), relying on the causal maskcausal mask, n.: A rule inside text-generating AI that stops each word from seeing the words after it, so the model learns to predict what comes next without being able to cheat. to reveal position implicitly.
The learned positional embeddings stop at 512, so longer inputs must be truncated.
Origin
'Position encoding' appears in end-to-end memory networks (Sainbayar Sukhbaatar and colleagues, 2015), and learned position embeddings in convolutional sequence-to-sequence models (Jonas Gehring and colleagues, 2017). 'Attention Is All You Need' (2017) introduced the sinusoidal scheme above; Peter Shaw, Jakob Uszkoreit and Ashish Vaswani proposed relative position representations in 2018.
Usage note
'Encoding' and 'embedding' are used loosely: strictly, an encoding is a fixed function and an embedding is learned.
Related
- rotary positional embeddingrotary positional embedding, n.: A way of telling a transformer where each word is by rotating parts of its internal numbers by an angle that grows with position, so the model can read off how far apart two words are.
- ALiBiALiBi, abbr.: A simple way to give a transformer a sense of word order: attention between two words is penalized more the farther apart they are, which helps it handle texts longer than those it trained on.
- context extensioncontext extension, n.: Techniques for stretching how much text a trained AI model can handle at once, usually by adjusting how it represents word positions and then training it briefly on longer examples.
- transformertransformer, n.: The design behind today's chatbots and much of modern AI. It handles text by letting every word look at every other word and decide which of them matter for understanding it.
- self-attentionself-attention, n.: Attention in which a passage looks at itself: every word checks every other word in the same text to work out its meaning in context, such as which noun 'it' refers to.
- embeddingembedding, n.: A list of numbers that stands for a word, picture or other item, arranged so that similar items get similar lists. It lets a computer measure similarity of meaning as distance.
Sources
- Vaswani, A. et al. (2017). Attention Is All You Need. NIPS 2017. arxiv.org/abs/1706.03762
- Shaw, P., Uszkoreit, J. & Vaswani, A. (2018). Self-Attention with Relative Position Representations. NAACL 2018. arxiv.org/abs/1803.02155
Mentioned in
context windowcontext window, n.: How much an AI model can read and keep in view at once: the instructions, the conversation so far and any attached documents. Text that does not fit has to be cut, summarized or left out., embeddingembedding, n.: A list of numbers that stands for a word, picture or other item, arranged so that similar items get similar lists. It lets a computer measure similarity of meaning as distance., transformertransformer, n.: The design behind today's chatbots and much of modern AI. It handles text by letting every word look at every other word and decide which of them matter for understanding it., long contextlong context, n.: The ability of an AI model to work with very long inputs — whole books, codebases or hours of transcripts — in one go, and the techniques that make this possible., neural radiance fieldneural radiance field, n.: A way of capturing a real scene from a set of photos as a small neural network that can then show the scene from any new viewpoint, as if moving a camera through it., vision transformervision transformer, n.: A way of applying the transformer, the design behind chatbots, to pictures: the image is cut into small square tiles that the model reads like words in a sentence., ALiBiALiBi, abbr.: A simple way to give a transformer a sense of word order: attention between two words is penalized more the farther apart they are, which helps it handle texts longer than those it trained on., ConformerConformer, prop. n.: A neural-network design for speech recognition that combines a transformer, which relates distant parts of a recording, with convolutions, which pick up fine local sound patterns., context extensioncontext extension, n.: Techniques for stretching how much text a trained AI model can handle at once, usually by adjusting how it represents word positions and then training it briefly on longer examples., equivarianceequivariance, n.: A property of a model whose output changes in step with its input: move the cat in a photo, and the network's internal 'cat' signal moves the same way., Fourier transformFourier transform, n.: A way of breaking a signal — a sound, an image, any pattern over time or space — into the simple waves of different frequencies that add up to it, much as a prism splits white light into colors., Jakob UszkoreitJakob Uszkoreit, prop. n.: A German computer scientist, one of the eight authors of the transformer paper, credited with first proposing to build language models on attention alone., patch embeddingpatch embedding, n.: The first step of a vision transformer: cutting an image into small square tiles and turning each tile into a list of numbers that the model can process like a word., rotary positional embeddingrotary positional embedding, n.: A way of telling a transformer where each word is by rotating parts of its internal numbers by an angle that grows with position, so the model can read off how far apart two words are., Transformer-XLTransformer-XL, prop. n.: A 2019 extension of the transformer that lets a language model carry a memory of earlier chunks of text forward, so it can follow connections longer than a single chunk of input.