ELMo

A method (Embeddings from Language Models) that produced contextual word representations from a pretrained bidirectional language model: two stacked LSTMlong short-term memory, n.: A type of recurrent network with a separate memory lane and learned 'gates' that decide what to remember, forget and output, letting it keep track of information over long sequences. networks, one reading left to right and one right to left over character-based word inputs, trained to predict the next and the previous word on about a billion words of text. For a downstream task, the outputs of all the model's layers are mixed with weights learned for that task and supplied as additional input features to an otherwise unchanged task model. Because each vector depends on the whole sentence, ELMo separated senses that static word embeddingsword embedding, n.: A list of numbers standing for a word, learned so that words used in similar ways get similar lists. It lets a computer measure that 'cat' is closer to 'dog' than to 'car'. conflate, and adding it set new best results on six benchmarks covering sentiment, named entities, coreference, semantic roles, entailment and question answering. Fine-tuned transformers such as BERTBERT, prop. n.: An influential 2018 Google language model that learned to understand text by filling in hidden words. It made search engines and many language tools much better at grasping meaning in context. and GPTGPT, abbr.: Short for 'generative pre-trained transformer', the family of OpenAI models behind ChatGPT. The letters are also used loosely for any similar chatbot model. soon superseded it, but it showed that pretrained language-model representations transfer broadly.

Adding ELMo vectors to our tagger raised F1 by two points in 2018.