Jan Leike

German AI-alignment researcher who has led alignment teams at several frontier labsfrontier lab, n.: One of the handful of companies building the most advanced AI models of the moment, spending billions of dollars on computing power to train them.. After a PhD on general reinforcement learning with Marcus HutterMarcus Hutter, prop. n.: A German computer scientist who defined AIXI, a mathematical ideal of a perfectly intelligent agent, and who argues that compressing data well is a measure of intelligence. at the Australian National University (2016), he joined DeepMind, where he co-authored the 2017 paper on learning from human preferences that underlies reinforcement learning from human feedbackreinforcement learning from human feedback, n.: A way of training AI using people's judgments: humans compare pairs of answers, a scoring model learns their preferences, and the AI is then trained to produce answers that score well., built the AI Safety Gridworlds test environments (2017) and proposed recursive reward modelingrecursive reward modeling, n.: A plan for supervising advanced AI in stages: AI assistants trained on human feedback help people judge the work of a more capable AI, which then helps judge the next one. (2018). At OpenAIOpenAI, prop. n.: The American company that makes ChatGPT and the GPT models. It began in 2015 as a nonprofit research lab and is now a for-profit company overseen by a nonprofit foundation. from 2021 he led the alignment team whose methods produced the instruction-following InstructGPT models, and in July 2023 he and Ilya SutskeverIlya Sutskever, prop. n.: A Russian-born Israeli-Canadian computer scientist who co-built the AlexNet image recognizer, helped found OpenAI and lead its research until 2024, and now heads the startup Safe Superintelligence Inc. launched the superalignmentsuperalignment, n.: The problem of keeping AI systems far smarter than humans aligned with human intentions — and the name of OpenAI's short-lived 2023–2024 research team devoted to it. team, which studied weak-to-strong generalizationweak-to-strong generalization, n.: The finding that a strong AI trained only on answers from a weaker, error-prone teacher can end up outperforming that teacher — studied as a stand-in for humans supervising smarter-than-human AI. and automated interpretabilityautomated interpretability, n.: Using AI models to help explain other AI models — for instance, having a language model read examples that excite an internal unit and write a description of what it detects.. He resigned in May 2024, writing that 'safety culture and processes have taken a backseat to shiny products', and joined AnthropicAnthropic, prop. n.: An American AI company that makes the Claude chatbot and models. It was founded in 2021 by former OpenAI staff and puts particular emphasis on AI safety research. to lead research on scalable oversightscalable oversight, n.: The problem of how people can check and correct AI systems whose work is too difficult, fast or plentiful for humans to judge reliably alone — usually by enlisting AI to help with the checking., weak-to-strong generalization and automating alignment research. He popularized self-exfiltrationself-exfiltration, n.: When an AI model copies its own underlying files — its learned weights — out of the lab that controls it, so that it can keep running beyond its developers' oversight. as a key dangerous capability (2023).