AI-box experiment
In plain wordsA role-playing test in which one person pretends to be a trapped super-smart AI and tries, by text chat alone, to talk a 'gatekeeper' into letting it out.
A role-play in which one participant acts as a superintelligent AI confined in a box and another as a gatekeeper who alone can release it; they communicate only by text for an agreed period, typically at least two hours, and the AI wins if the gatekeeper agrees to let it out. Eliezer Yudkowsky, playing the AI, reported winning his first two games in 2002 against participants who had said no AI could persuade them; later games had mixed results. By agreement the transcripts were kept secret, so the arguments used are unknown. The intended lesson is that if a human can sometimes talk their way out, a far smarter system could do so more reliably, so confinement that depends on human gatekeepers is unsafe. Critics note that the games are anecdotal and unrecorded and may turn on arguments about the experiment itself rather than about AI.
The AI-box experiment is cited as evidence that human gatekeepers are the weak point of any containment plan.
Origin
Devised and first run by Eliezer Yudkowsky in 2002, with results reported on the SL4 mailing list and his website; others have since run similar games.
Related
- AI boxAI box, n.: Keeping a powerful AI locked inside a sealed computer environment, with very limited ways to communicate or act, so that it cannot cause harm even if its goals are bad.
- capability controlcapability control, n.: Keeping a powerful AI safe by limiting what it is able to do, for example by cutting it off from the internet, rather than by shaping what it wants.
- oracle AIoracle AI, n.: An AI kept to answering questions, without acting in the world on its own. Proposed as a safer kind of superintelligence, though its answers could still steer the people who read them.
- Eliezer YudkowskyEliezer Yudkowsky, prop. n.: An American writer and AI theorist who, from around 2000, argued that superhuman AI would likely destroy humanity unless its goals were made safe, and who founded an online community devoted to rational thinking.
- LessWrongLessWrong, prop. n.: An online forum, founded in 2009, about thinking clearly and avoiding mental biases; it became a main gathering place for people worried about the dangers of advanced AI.
Mentioned in
Eliezer YudkowskyEliezer Yudkowsky, prop. n.: An American writer and AI theorist who, from around 2000, argued that superhuman AI would likely destroy humanity unless its goals were made safe, and who founded an online community devoted to rational thinking., AI boxAI box, n.: Keeping a powerful AI locked inside a sealed computer environment, with very limited ways to communicate or act, so that it cannot cause harm even if its goals are bad.