Transformer-XL

A transformertransformer, n.: The design behind today's chatbots and much of modern AI. It handles text by letting every word look at every other word and decide which of them matter for understanding it. language modellanguage model, n.: A system that has learned how likely different sequences of words are, so it can predict the next word in a text and, by doing that repeatedly, write text of its own. architecture that adds segment-level recurrence: hidden states computed for the previous segment of text are cached and reused, without gradient flow, as extra context when the next segment is processed, so the usable context extends across segments and grows with depth. Because reused states come from different absolute positions, it introduced a relative positional encodingpositional encoding, n.: Extra information added to each word so a transformer knows where it sits in the text. Without it, 'dog bites man' and 'man bites dog' would look the same to the model. in the attention scores that influenced later position schemes. The authors reported dependencies about 80% longer than recurrent networks and 450% longer than vanilla transformers could learn, record results at the time on five language-modeling benchmarks (for example perplexity 18.3 on WikiText-103), and evaluation up to 1,800 times faster than a vanilla transformer. Its cached memory anticipated the KV cacheKV cache, n.: A memory store where a chatbot keeps its working notes on everything already read and written, so each new word does not require re-reading the whole conversation from scratch. and streaming methods of later long contextlong context, n.: The ability of an AI model to work with very long inputs — whole books, codebases or hours of transcripts — in one go, and the techniques that make this possible. work, and it was the backbone of XLNet (2019).

Transformer-XL's cached memory let us evaluate on whole Wikipedia articles instead of 512-token chunks.