Jakob Uszkoreit

German computer scientist and a co-author of 'Attention Is All You NeedAttention Is All You Need, prop. n.: The 2017 Google research paper that introduced the transformer, the neural-network design behind ChatGPT, Claude, Gemini and nearly every other modern AI language system.' (2017), whose contribution notes credit him with proposing to replace recurrence with self-attentionself-attention, n.: Attention in which a passage looks at itself: every word checks every other word in the same text to work out its meaning in context, such as which noun 'it' refers to. and with starting the effort to test the idea, which became the transformertransformer, n.: The design behind today's chatbots and much of modern AI. It handles text by letting every word look at every other word and decide which of them matter for understanding it.. At Google from 2008 he worked on machine translation and language understanding, and with colleagues he extended attention-based models beyond the original design: relative position representations (2018, see positional encodingpositional encoding, n.: Extra information added to each word so a transformer knows where it sits in the text. Without it, 'dog bites man' and 'man bites dog' would look the same to the model.), blockwise parallel decoding (2018), a precursor of speculative decodingspeculative decoding, n.: A speed-up trick in which a small, fast model guesses the next few words and the big model checks all the guesses at once, keeping the right ones: faster answers, same quality. and multi-token predictionmulti-token prediction, n.: Training a language model to guess several upcoming words at once rather than only the next one, which can help it learn better and later write faster., the Image Transformer (2018) and the vision transformervision transformer, n.: A way of applying the transformer, the design behind chatbots, to pictures: the image is cut into small square tiles that the model reads like words in a sentence. (2020). In 2021 he co-founded Inceptive, a company that uses deep learning to design RNA molecules for medicines, which he leads as chief executive.