moral crumple zone
In plain wordsWhen a largely automated system goes wrong and the human who was supposed to supervise it gets blamed, even though they had little real say; named after the car part that soaks up a crash.
The position a human operator can be left in when a partly automated system fails: blamed, morally and legally, for an outcome the automation largely determined and the person had little power to prevent. The image inverts the car part it borrows from: a crumple zone deforms to shield the people inside, whereas here a person takes the blame and the system and its makers come away unscathed. Cases include pilots blamed after confusing autopilot handoffs and safety drivers blamed in self-driving car crashes. The concept warns that human-in-the-loophuman-in-the-loop, adj.: Describes AI systems in which a person checks, corrects or approves the machine's work at key points, rather than letting it act entirely on its own. designs can create a nominal overseer who lacks the time, information or authority to intervene, and it argues for allocating responsibility according to actual control and for designing oversight that is meaningful rather than symbolic.
Putting a reviewer on every AI decision without time to check them just builds a moral crumple zone.
Origin
Coined by anthropologist Madeleine Clare Elish in a paper first presented at the We Robot conference in 2016 and published as 'Moral Crumple Zones: Cautionary Tales in Human-Robot Interaction' (Engaging Science, Technology, and Society, 2019), drawing on cases such as the 2009 loss of Air France Flight 447.
Related
- responsibility gapresponsibility gap, n.: The problem that when an AI system that learns or acts on its own causes harm, it may be unclear whether anyone (its makers, users or operators) can fairly be held responsible.
- automation biasautomation bias, n.: The habit of trusting a computer's suggestion too much: following it even when it is wrong, or missing problems because the machine did not flag them.
- human oversighthuman oversight, n.: Keeping people able to watch, understand, question and, if necessary, stop or overrule an AI system, rather than letting it act unchecked.
- human-in-the-loophuman-in-the-loop, adj.: Describes AI systems in which a person checks, corrects or approves the machine's work at key points, rather than letting it act entirely on its own.
- AI liabilityAI liability, n.: Who has to pay or answer in court when an AI system causes harm — the company that built it, the business that used it or the person who relied on it — and under which laws.
- deskillingdeskilling, n.: The loss of people's skills when machines take over the work that used to keep those skills sharp, or when jobs are redesigned so that less skill is needed.
Sources
- Elish, M. C. (2019). Moral Crumple Zones: Cautionary Tales in Human-Robot Interaction. Engaging Science, Technology, and Society 5, 40–60.
Mentioned in
automation biasautomation bias, n.: The habit of trusting a computer's suggestion too much: following it even when it is wrong, or missing problems because the machine did not flag them., deskillingdeskilling, n.: The loss of people's skills when machines take over the work that used to keep those skills sharp, or when jobs are redesigned so that less skill is needed., responsibility gapresponsibility gap, n.: The problem that when an AI system that learns or acts on its own causes harm, it may be unclear whether anyone (its makers, users or operators) can fairly be held responsible.