EleutherAI

An open-science institute working on large language modelslarge language model, n.: A computer program trained on enormous amounts of text to predict the words that come next. At vast scale, that simple skill lets it chat, answer questions, translate, summarize and write code. — how to train them, how they work inside (interpretabilityinterpretability, n.: The effort to understand how an AI system arrives at its outputs — what information it uses and how it combines it — instead of treating it as a sealed black box.) and how to align them — that publishes its models, data and code; it began as an informal volunteer collective and became a nonprofit in 2023. In 2020–2022 it assembled The PileThe Pile, prop. n.: A large free collection of English text released in 2020 by the volunteer group EleutherAI, mixing 22 sources from books and science papers to code, used to train early open language models., an 825-gigabyte English text dataset, and released GPT-Neo, GPT-J and GPT-NeoX-20B, among the first openly downloadable GPT-3-style models, at a time when large models were held almost exclusively by companies. Its Pythia suite (2023), a set of models trained on identical data with public intermediate checkpoints, became a standard tool for studying training dynamics and memorization, and its lm-evaluation-harness is one of the most widely used evaluation harnessesevaluation harness, n.: Software that puts AI models through tests in a fixed, repeatable way — feeding in the questions, collecting the answers and scoring them — so that every model is judged identically.. The inclusion in The Pile of Books3, a collection of pirated books, drew it into copyright litigation; in 2025 it co-released the Common Pile, a large dataset of openly licensed and public-domain text.