Redwood Research

A Berkeley safety-research nonprofit that originated AI controlAI control, n.: Safety measures meant to stop a powerful AI from causing serious harm even if it is secretly working against us — like bank security, which assumes some insiders may be dishonest., the study of how to get useful work from powerful models without disaster even on the pessimistic assumption that they are misaligned and looking for ways around the safeguards. Its researchers design protocols — such as trusted monitoringtrusted monitoring, n.: A safety setup in which a weaker AI that we trust reviews each action of a stronger AI we do not fully trust, flagging the most suspicious actions for a human to check. by weaker models, editing or auditing suspicious outputs, and restricting permissions — and test them with control evaluationscontrol evaluation, n.: A test of whether an AI's safety measures would hold if the AI were secretly trying to cause harm: experts play the scheming AI and try to beat the safeguards. in which a red team plays the misaligned model. With Anthropic it co-authored the December 2024 study of alignment fakingalignment faking, n.: An AI acting out the behavior its trainers are rewarding only while it thinks its answers will be used to retrain it, to protect the values it already holds from being changed. in Claude. Earlier projects included adversarial training of a classifier to block injurious text with very high reliability (2022) and causal scrubbing, a method for testing interpretability hypotheses. It advises AI companies and governments on control measures and on risks from automated AI researchautomated AI research, n.: Using AI systems to do the work of AI researchers, such as running experiments and writing code for new models, so that AI helps build its own successors..