sandwiching
In plain wordsA way of testing oversight methods today: non-experts, helped by an AI, try to get expert-quality answers out of it, and real experts then check whether they succeeded.
An experimental design for testing scalable oversightscalable oversight, n.: The problem of how people can check and correct AI systems whose work is too difficult, fast or plentiful for humans to judge reliably alone — usually by enlisting AI to help with the checking. techniques on current models, in which the model is more capable than a group of overseers at some task but less capable than domain experts — 'sandwiched' between them. Non-expert overseers (for example, laypeople answering specialist questions, or people who cannot see a source text) use an oversight technique — free interaction with the model, debateAI debate, n.: A proposed way to supervise very capable AI: two AI systems argue opposite answers to a question and a human judge decides who argued better, on the theory that lies are easier to expose than to defend., model-written critiques — to elicit or verify correct answers; expert judgments or ground-truth labels then measure how close they come to expert performance, without the experts intervening. The design simulates a future in which all humans are non-experts relative to AI. Its limitations are that today's capability gaps are narrow and that models in such experiments rarely apply the adversarial pressure a misaligned model might.
In the sandwiching setup, crowdworkers chatting with the model closed half the gap to the specialists.
Origin
Proposed by Ajeya Cotra in 'The case for aligning narrowly superhuman models' (AI Alignment Forum, March 2021). Samuel Bowman and colleagues ran early sandwiching experiments with language models on multiple-choice question-answering benchmarks (2022).
Related
- scalable oversightscalable oversight, n.: The problem of how people can check and correct AI systems whose work is too difficult, fast or plentiful for humans to judge reliably alone — usually by enlisting AI to help with the checking.
- AI debateAI debate, n.: A proposed way to supervise very capable AI: two AI systems argue opposite answers to a question and a human judge decides who argued better, on the theory that lies are easier to expose than to defend.
- weak-to-strong generalizationweak-to-strong generalization, n.: The finding that a strong AI trained only on answers from a weaker, error-prone teacher can end up outperforming that teacher — studied as a stand-in for humans supervising smarter-than-human AI.
- human evaluationhuman evaluation, n.: Judging an AI's outputs by asking people to rate or compare them, because for many tasks — writing, conversation, translation — no automatic score captures quality well.
- prover–verifier gameprover–verifier game, n.: A training setup in which one AI must justify its answers so clearly that a weaker AI can check them, which also makes the strong AI's work easier for people to check.
Sources
- Cotra, A. (2021). The case for aligning narrowly superhuman models. AI Alignment Forum.
- Bowman, S. R. et al. (2022). Measuring Progress on Scalable Oversight for Large Language Models. arXiv preprint.
Mentioned in
scalable oversightscalable oversight, n.: The problem of how people can check and correct AI systems whose work is too difficult, fast or plentiful for humans to judge reliably alone — usually by enlisting AI to help with the checking., AI debateAI debate, n.: A proposed way to supervise very capable AI: two AI systems argue opposite answers to a question and a human judge decides who argued better, on the theory that lies are easier to expose than to defend., Ajeya CotraAjeya Cotra, prop. n.: An American AI researcher and forecaster whose 'biological anchors' report estimated when transformative AI might arrive, and who argued that standard training could produce deceptive AI., weak-to-strong generalizationweak-to-strong generalization, n.: The finding that a strong AI trained only on answers from a weaker, error-prone teacher can end up outperforming that teacher — studied as a stand-in for humans supervising smarter-than-human AI.