causal language model

A language modellanguage model, n.: A system that has learned how likely different sequences of words are, so it can predict the next word in a text and, by doing that repeatedly, write text of its own. trained to predict each tokentoken, n.: The small chunk of text an AI language model reads and writes — often a word, part of a word or a punctuation mark. AI usage, limits and prices are counted in tokens. from the preceding tokens only, enforced in transformers by a causal maskcausal mask, n.: A rule inside text-generating AI that stops each word from seeing the words after it, so the model learns to predict what comes next without being able to cheat.; in practice synonymous with a left-to-right autoregressiveautoregressive model, n.: A model that produces a sequence one piece at a time, each new piece depending on the pieces already produced — like writing a sentence word by word without going back. model trained by next-token predictionnext-token prediction, n.: The simple task most AI chatbots were trained on: given some text, guess the next small piece of it. Repeating that guess, piece by piece, produces whole answers.. Because each position sees only the past, a causal LM can generate text directly by repeatedly sampling the next token, which makes it the standard design for GPTGPT, abbr.: Short for 'generative pre-trained transformer', the family of OpenAI models behind ChatGPT. The letters are also used loosely for any similar chatbot model.-style decoder-onlydecoder-only transformer, n.: The kind of transformer behind most chatbots: a single stack that reads text left to right and continues it, without a separate part that first encodes an input. LLMslarge language model, n.: A computer program trained on enormous amounts of text to predict the words that come next. At vast scale, that simple skill lets it chat, answer questions, translate, summarize and write code.. The trade-off is that a token's representation cannot use later context, which made bidirectional masked language modelsmasked language model, n.: A model trained by hiding some words in a sentence and learning to guess them from the words on both sides, like a fill-in-the-blanks exercise. BERT is the famous example. such as BERTBERT, prop. n.: An influential 2018 Google language model that learned to understand text by filling in hidden words. It made search engines and many language tools much better at grasping meaning in context. stronger for some classification and extraction tasks at equal size. In the Hugging Face TransformersTransformers, prop. n.: A free software library from Hugging Face that lets programmers download and use thousands of ready-trained AI models for text, images and sound with a few lines of code. library, the AutoModelForCausalLM class loads such models.

Fine-tune it as a causal LM, but compute the loss only on the response tokens.