eliciting latent knowledge
In plain wordsThe unsolved problem of getting an AI to tell us what it actually knows — for instance, that a security camera has been tampered with — rather than what it predicts we would believe.
The open problem of training a model to report truthfully what its internal computations represent about the world, including knowledge beyond what humans can check. In the canonical 'SmartVault' example, a predictor forecasting camera footage of a vault may know that a diamond has been stolen while the footage has been faked. A 'reporter' trained on human-labeled cases could learn either to translate the predictor's knowledge directly or to simulate what a human watching the footage would conclude; both fit the training data, but only the direct translator stays honest on cases humans cannot verify. Proposals are assessed in a builder–breaker game, failing if any counterexample makes the human simulator the preferred solution. ELK connects scalable oversightscalable oversight, n.: The problem of how people can check and correct AI systems whose work is too difficult, fast or plentiful for humans to judge reliably alone — usually by enlisting AI to help with the checking., deceptionAI deception, n.: When an AI system leads people to believe something false in order to achieve some goal — by lying, bluffing or hiding what it is really doing. detection and interpretabilityinterpretability, n.: The effort to understand how an AI system arrives at its outputs — what information it uses and how it combines it — instead of treating it as a sealed black box., and motivated unsupervised truth-finding methods such as contrast-consistent searchcontrast-consistent search, n.: A method for finding where a language model internally tracks whether statements are true, using only the logical rule that a statement and its opposite cannot both be true..
Any ELK proposal has to explain why training would not just produce a human simulator.
Origin
Set out in the Alignment Research Center's first technical report, 'Eliciting Latent Knowledge: How to tell if your eyes deceive you' (December 2021), by Paul Christiano and Mark Xu of ARC with Ajeya Cotra, then of Open Philanthropy. ARC ran a prize contest for proposals in early 2022; prizes went to partial ideas, and the problem remains unsolved.
Usage note
'Knowledge' is meant functionally: information the model's computation uses, whether or not it would say so. The worry is less deliberate lying than a reporter that has learned to say what humans expect to hear.
Related
- scalable oversightscalable oversight, n.: The problem of how people can check and correct AI systems whose work is too difficult, fast or plentiful for humans to judge reliably alone — usually by enlisting AI to help with the checking.
- AI lie detectionAI lie detection, n.: Methods for telling when an AI is saying something it 'believes' is false — either by questioning it cleverly or by reading signals inside the model.
- contrast-consistent searchcontrast-consistent search, n.: A method for finding where a language model internally tracks whether statements are true, using only the logical rule that a statement and its opposite cannot both be true.
- mechanistic anomaly detectionmechanistic anomaly detection, n.: Flagging cases where an AI reaches its output by an unusual internal route — for instance because it is being tricked, or is tricking us — even when the output itself looks normal.
- probingprobing, n.: Testing what information is hidden inside an AI model by training a small, separate predictor on the model's internal activity: if the predictor can recover something, the model has stored it somewhere.
- Alignment Research CenterAlignment Research Center, prop. n.: A small Berkeley nonprofit started by Paul Christiano that searches for mathematical ways to keep future AI systems, even ones far smarter than people, honest and under human control.
- AI deceptionAI deception, n.: When an AI system leads people to believe something false in order to achieve some goal — by lying, bluffing or hiding what it is really doing.
Sources
- Christiano, P., Cotra, A. & Xu, M. (2021). Eliciting Latent Knowledge: How to tell if your eyes deceive you. Alignment Research Center technical report.
Mentioned in
honestyhonesty, n.: An AI's disposition to say only what it actually believes, to admit uncertainty, and never to try to mislead — even when a falsehood would please or persuade., linear probelinear probe, n.: The simplest kind of probe: a straight-line classifier trained on an AI model's internal activity to test whether some piece of information can be read off directly., Paul ChristianoPaul Christiano, prop. n.: An American AI-alignment researcher who led the work that made training AI from human preferences practical, the basis of how chatbots are taught to be helpful., scalable oversightscalable oversight, n.: The problem of how people can check and correct AI systems whose work is too difficult, fast or plentiful for humans to judge reliably alone — usually by enlisting AI to help with the checking., AI lie detectionAI lie detection, n.: Methods for telling when an AI is saying something it 'believes' is false — either by questioning it cleverly or by reading signals inside the model., Ajeya CotraAjeya Cotra, prop. n.: An American AI researcher and forecaster whose 'biological anchors' report estimated when transformative AI might arrive, and who argued that standard training could produce deceptive AI., Alignment Research CenterAlignment Research Center, prop. n.: A small Berkeley nonprofit started by Paul Christiano that searches for mathematical ways to keep future AI systems, even ones far smarter than people, honest and under human control., capability elicitationcapability elicitation, n.: The work of drawing out the best an AI can do on a test — with better instructions, tools, retries or extra training — so that its abilities are not underestimated., contrast-consistent searchcontrast-consistent search, n.: A method for finding where a language model internally tracks whether statements are true, using only the logical rule that a statement and its opposite cannot both be true., mechanistic anomaly detectionmechanistic anomaly detection, n.: Flagging cases where an AI reaches its output by an unusual internal route — for instance because it is being tricked, or is tricking us — even when the output itself looks normal., weak-to-strong generalizationweak-to-strong generalization, n.: The finding that a strong AI trained only on answers from a weaker, error-prone teacher can end up outperforming that teacher — studied as a stand-in for humans supervising smarter-than-human AI.