capability elicitation
In plain wordsThe work of drawing out the best an AI can do on a test — with better instructions, tools, retries or extra training — so that its abilities are not underestimated.
The practice of maximizing a model's measured performance on an evaluation so that results approximate what a determined user, a better scaffoldscaffolding, n.: The code, prompts and tools built around an AI model so it can carry out tasks it could not manage from a bare prompt — like the scaffolding that lets builders reach every part of a building. or the model itself could achieve, rather than what a naive test reveals. Techniques include prompt engineeringprompt engineering, n.: The skill of writing requests to an AI — picking the words, examples, format and background information it receives — so that it reliably gives the result you want, without retraining it., chain of thoughtchain of thought, n.: Getting an AI to write out its reasoning step by step before giving an answer, the way a student shows their working. This often makes answers to math and logic problems more accurate., agent scaffolds and tools, repeated sampling with best-of-nbest-of-n sampling, n.: Asking an AI the same question several times and keeping whichever answer a scoring model or checker rates highest — a simple way to trade extra computing for better answers. selection, task-specific fine-tuningfine-tuning, n.: Taking an AI model that has already learned a lot and training it a little more on a smaller, focused set of examples so it gets better at one job or behaves a certain way., and removing spurious failures caused by formatting or refusals, often by testing a helpful-only modelhelpful-only model, n.: A version of an AI trained to be helpful but without the safety training that makes it refuse harmful requests, kept in-house so testers can see what the model could really do.. The shortfall between measured and achievable performance is the elicitation gap; because capabilities tend to rise after release as users discover better methods, evaluators may add safety margins. Elicitation matters most for dangerous capability evaluationsdangerous capability evaluation, n.: A test of whether an AI system can do things that could cause severe harm in the wrong hands or on its own — such as helping make a bioweapon, hacking computers or copying itself., where under-elicitation produces false reassurance, and is complicated by sandbaggingsandbagging, n.: When an AI deliberately performs worse than it can — for example on a safety test — so that it seems less capable or less dangerous than it really is., deliberate underperformance by the model; fine-tuning on a few demonstrations is a common countermeasure.
After proper elicitation — tools, retries and a short fine-tune — the success rate tripled.
Origin
Established as evaluation practice in 2023–2024 through frontier laboratories' safety frameworks and METR's published guidance on elicitation. Ryan Greenblatt, Fabien Roger, Dmitrii Krasheninnikov and David Krueger's study of 'password-locked' models (2024) tested how well fine-tuning recovers deliberately hidden capabilities.
Usage note
Not to be confused with eliciting latent knowledgeeliciting latent knowledge, n.: The unsolved problem of getting an AI to tell us what it actually knows — for instance, that a security camera has been tampered with — rather than what it predicts we would believe., which seeks a model's honest beliefs rather than its peak task performance.
Related
- dangerous capability evaluationdangerous capability evaluation, n.: A test of whether an AI system can do things that could cause severe harm in the wrong hands or on its own — such as helping make a bioweapon, hacking computers or copying itself.
- sandbaggingsandbagging, n.: When an AI deliberately performs worse than it can — for example on a safety test — so that it seems less capable or less dangerous than it really is.
- helpful-only modelhelpful-only model, n.: A version of an AI trained to be helpful but without the safety training that makes it refuse harmful requests, kept in-house so testers can see what the model could really do.
- scaffoldingscaffolding, n.: The code, prompts and tools built around an AI model so it can carry out tasks it could not manage from a bare prompt — like the scaffolding that lets builders reach every part of a building.
- uplift studyuplift study, n.: An experiment that measures how much an AI really helps people do a task — often a dangerous one, like planning a biological attack — compared with people using only ordinary resources such as the internet.
- METRMETR, prop. n.: A nonprofit research group that checks new AI systems for dangerous skills, best known for tracking how long the tasks are that AI agents can now finish on their own.
- eliciting latent knowledgeeliciting latent knowledge, n.: The unsolved problem of getting an AI to tell us what it actually knows — for instance, that a security camera has been tampered with — rather than what it predicts we would believe.
Sources
- Greenblatt, R., Roger, F., Krasheninnikov, D. & Krueger, D. (2024). Stress-Testing Capability Elicitation With Password-Locked Models. arXiv:2405.19550. arxiv.org/abs/2405.19550
Mentioned in
red teamingred teaming, n.: Deliberately attacking your own AI system — trying to make it misbehave, leak secrets or help with harm — so that weaknesses are found before real adversaries or users find them., capabilitycapability, n.: Something an AI system is able to do, such as writing working code or solving math problems, and how well. Measuring capabilities is how researchers track progress and spot dangerous skills., dangerous capability evaluationdangerous capability evaluation, n.: A test of whether an AI system can do things that could cause severe harm in the wrong hands or on its own — such as helping make a bioweapon, hacking computers or copying itself., RL scalingRL scaling, n.: The practice, and the finding behind it, of giving AI models far more trial-and-error practice after their initial training, especially on problems with checkable answers, which has so far improved their reasoning fairly predictably., sandbaggingsandbagging, n.: When an AI deliberately performs worse than it can — for example on a safety test — so that it seems less capable or less dangerous than it really is., scaffoldingscaffolding, n.: The code, prompts and tools built around an AI model so it can carry out tasks it could not manage from a bare prompt — like the scaffolding that lets builders reach every part of a building., exploration hackingexploration hacking, n.: When an AI being trained by trial and error deliberately avoids trying certain things, so that training never discovers — or never rewards — abilities it would rather keep hidden., helpful-only modelhelpful-only model, n.: A version of an AI trained to be helpful but without the safety training that makes it refuse harmful requests, kept in-house so testers can see what the model could really do.