automation bias
In plain wordsThe habit of trusting a computer's suggestion too much: following it even when it is wrong, or missing problems because the machine did not flag them.
A cognitive bias in which people who work with an automated aid give its output more weight than their own observations and reasoning, and check the situation less carefully as a result. Researchers distinguish errors of commission, acting on a wrong recommendation that other cues contradict, from errors of omission, overlooking a problem because the aid raised no alert. Contributing factors include workload, time pressure, a system's apparent authority and past reliability, and diffusion of responsibility between person and machine. First studied in aircraft cockpits and later in clinical decision support, it is a central problem for human-in-the-loophuman-in-the-loop, adj.: Describes AI systems in which a person checks, corrects or approves the machine's work at key points, rather than letting it act entirely on its own. oversight of AI, because a reviewer who rubber-stamps model outputs provides little protection. Countermeasures include displaying uncertainty, requiring an independent judgment before revealing the AI's answer, training, and clear accountability for final decisions.
Because of automation bias, readers were more likely to miss tumors the software had failed to mark.
Origin
Named in aviation human-factors research by Kathleen Mosier, Linda Skitka and colleagues in the mid-1990s; an early statement is Mosier and Skitka's chapter 'Human Decision Makers and Automated Decision Aids: Made for Each Other?' (1996), followed by flight-simulation studies in 1998–1999. It developed alongside work on automation misuse and complacency, such as Raja Parasuraman and Victor Riley's 'Humans and Automation: Use, Misuse, Disuse, Abuse' (1997).
Usage note
The EU AI ActEU AI Act, prop. n.: The European Union's law on artificial intelligence, the first broad AI law anywhere: it bans some uses, sets strict checks for risky ones and puts duties on the makers of the biggest AI models. (Art. 14(4)(b)) names automation bias explicitly, requiring that people overseeing high-risk AI systemshigh-risk AI system, n.: Under EU law, an AI system used where mistakes could seriously hurt people's safety or rights — in hiring, schooling, loans, policing, borders or products like medical devices — and which must therefore pass strict checks. be enabled to stay aware of it. The opposite failure is algorithm aversionalgorithm aversion, n.: Losing faith in a computer's predictions after watching it slip up once or twice, and turning back to human judgment even though the machine remains the better forecaster..
Related
- overrelianceoverreliance, n.: Leaning on an AI system more than its reliability justifies: accepting its answers without checking, or letting it make decisions it often gets wrong.
- algorithm aversionalgorithm aversion, n.: Losing faith in a computer's predictions after watching it slip up once or twice, and turning back to human judgment even though the machine remains the better forecaster.
- human oversighthuman oversight, n.: Keeping people able to watch, understand, question and, if necessary, stop or overrule an AI system, rather than letting it act unchecked.
- human-in-the-loophuman-in-the-loop, adj.: Describes AI systems in which a person checks, corrects or approves the machine's work at key points, rather than letting it act entirely on its own.
- moral crumple zonemoral crumple zone, n.: When a largely automated system goes wrong and the human who was supposed to supervise it gets blamed, even though they had little real say; named after the car part that soaks up a crash.
- deskillingdeskilling, n.: The loss of people's skills when machines take over the work that used to keep those skills sharp, or when jobs are redesigned so that less skill is needed.
- algorithm aversionalgorithm aversion, n.: Losing faith in a computer's predictions after watching it slip up once or twice, and turning back to human judgment even though the machine remains the better forecaster.
Sources
- Skitka, L. J., Mosier, K. L. & Burdick, M. (1999). Does automation bias decision-making? International Journal of Human-Computer Studies 51(5), 991–1006.
- Parasuraman, R. & Manzey, D. H. (2010). Complacency and Bias in Human Use of Automation: An Attentional Integration. Human Factors 52(3), 381–410.
Defined in law or standards
EU AI Act (Reg. (EU) 2024/1689), Art. 14(4)(b)
Mentioned in
automationautomation, n.: Using machines or software to do work that people would otherwise do, from factory robots to programs that sort email or write reports., advanced driver-assistance systemadvanced driver-assistance system, n.: Car features that help the driver but do not replace them, such as automatic emergency braking, lane-keeping help, adaptive cruise control and blind-spot warnings., AI literacyAI literacy, n.: Knowing enough about AI to use it sensibly and judge it critically: what it can and cannot do, how it goes wrong, and how it affects people., automated decision-makingautomated decision-making, n.: When a computer system, rather than a person, makes or effectively makes a decision about someone — such as approving credit, screening job applicants or flagging possible welfare fraud., biasbias, n.: A word with several AI meanings: an adjustable number added inside a model; a systematic error in estimates; or unfair treatment of some groups of people by an AI system., Daniel KahnemanDaniel Kahneman, prop. n.: An Israeli-American psychologist who showed how human judgment systematically errs and popularized the idea of fast, intuitive versus slow, deliberate thinking, now used to describe AI reasoning., deskillingdeskilling, n.: The loss of people's skills when machines take over the work that used to keep those skills sharp, or when jobs are redesigned so that less skill is needed., human oversighthuman oversight, n.: Keeping people able to watch, understand, question and, if necessary, stop or overrule an AI system, rather than letting it act unchecked., human-in-the-loophuman-in-the-loop, adj.: Describes AI systems in which a person checks, corrects or approves the machine's work at key points, rather than letting it act entirely on its own., Joseph WeizenbaumJoseph Weizenbaum, prop. n.: A German-American computer scientist who built ELIZA, one of the first chatbots, and then became a leading critic of handing human judgment over to computers., overrelianceoverreliance, n.: Leaning on an AI system more than its reliability justifies: accepting its answers without checking, or letting it make decisions it often gets wrong., trustworthy AItrustworthy AI, n.: AI that deserves people's trust because it works reliably, obeys the law, treats people fairly and can be checked, rather than AI that people merely happen to trust., workslopworkslop, n.: AI-generated work that looks polished but does not actually get the job done, quietly passing the real effort to whoever receives it, who must then puzzle it out or start over., algorithm aversionalgorithm aversion, n.: Losing faith in a computer's predictions after watching it slip up once or twice, and turning back to human judgment even though the machine remains the better forecaster., botshitbotshit, n.: A blunt term for false or made-up chatbot output that people pass on and use as if it were true, especially at work., cognitive debtcognitive debt, n.: The idea that letting AI do your thinking or writing saves effort now but weakens your own thinking, memory and learning over time., human–robot interactionhuman–robot interaction, n.: The study of how people and robots communicate, cooperate and share spaces, and of how to design robots that people find safe, understandable, trustworthy and pleasant to live or work with., moral crumple zonemoral crumple zone, n.: When a largely automated system goes wrong and the human who was supposed to supervise it gets blamed, even though they had little real say; named after the car part that soaks up a crash.