contrast-consistent search
In plain wordsA method for finding where a language model internally tracks whether statements are true, using only the logical rule that a statement and its opposite cannot both be true.
An unsupervised method for discovering a truth-like direction in a language model's activationsactivation, n.: The numbers a neural network computes inside itself as it processes an input — a reading of which of its artificial neurons are firing, and how strongly, at that moment. without labeled data. For each yes/no question, the model processes two contrasting statements — one answering 'yes', one 'no' — and a linear probelinear probe, n.: The simplest kind of probe: a straight-line classifier trained on an AI model's internal activity to test whether some piece of information can be read off directly. is trained on their normalized hidden states so that its two probabilities are consistent (summing to about one) and confident (not both near one-half). The original study found such probes recovered correct answers more accurately than the model's own outputs and stayed accurate when the model was prompted to answer wrongly, suggesting latent knowledge separable from what the model says. Later analysis showed that the consistency objective can also be satisfied by other salient features, such as a fictional character's opinion stated in the prompt, so CCS does not reliably isolate a model's own beliefs.
The CCS probe kept answering correctly even when the prompt nudged the model toward a wrong answer.
Origin
Introduced by Collin Burns, Haotian Ye, Dan Klein and Jacob Steinhardt in 'Discovering Latent Knowledge in Language Models Without Supervision' (December 2022; ICLR 2023). Sebastian Farquhar and colleagues at Google DeepMind published the main critique in 2023.
Usage note
Motivated by eliciting latent knowledgeeliciting latent knowledge, n.: The unsolved problem of getting an AI to tell us what it actually knows — for instance, that a security camera has been tampered with — rather than what it predicts we would believe.; it is one approach to AI lie detectionAI lie detection, n.: Methods for telling when an AI is saying something it 'believes' is false — either by questioning it cleverly or by reading signals inside the model., not a solution to it.
Related
- eliciting latent knowledgeeliciting latent knowledge, n.: The unsolved problem of getting an AI to tell us what it actually knows — for instance, that a security camera has been tampered with — rather than what it predicts we would believe.
- AI lie detectionAI lie detection, n.: Methods for telling when an AI is saying something it 'believes' is false — either by questioning it cleverly or by reading signals inside the model.
- linear probelinear probe, n.: The simplest kind of probe: a straight-line classifier trained on an AI model's internal activity to test whether some piece of information can be read off directly.
- probingprobing, n.: Testing what information is hidden inside an AI model by training a small, separate predictor on the model's internal activity: if the predictor can recover something, the model has stored it somewhere.
- representation engineeringrepresentation engineering, n.: An approach to understanding and controlling AI models by finding where high-level ideas such as honesty or danger show up in their internal activity, then reading or adjusting those signals directly.
- linear representation hypothesislinear representation hypothesis, n.: The idea that a neural network stores many concepts as simple directions inside its internal activity, so how much of a concept is present can be read off by measuring along its direction.
Sources
- Burns, C., Ye, H., Klein, D. & Steinhardt, J. (2023). Discovering Latent Knowledge in Language Models Without Supervision. ICLR 2023.
- Farquhar, S. et al. (2023). Challenges with unsupervised LLM knowledge discovery. Google DeepMind, arXiv preprint.
Mentioned in
AI lie detectionAI lie detection, n.: Methods for telling when an AI is saying something it 'believes' is false — either by questioning it cleverly or by reading signals inside the model., eliciting latent knowledgeeliciting latent knowledge, n.: The unsolved problem of getting an AI to tell us what it actually knows — for instance, that a security camera has been tampered with — rather than what it predicts we would believe.