algorithm aversion
In plain wordsLosing faith in a computer's predictions after watching it slip up once or twice, and turning back to human judgment even though the machine remains the better forecaster.
A preference for human judgment, one's own or an expert's, over a demonstrably more accurate algorithm, which strengthens once people have watched the algorithm make mistakes. In the incentivized forecasting experiments that named it, people who had seen a statistical model at work were less willing to rely on it than people who had not, even when it had clearly beaten the human alternative: an error cost the model more trust than the same error cost a person. Suggested causes include beliefs that humans learn from experience while algorithms cannot, intolerance of any error, a sense that algorithms ignore individual circumstances, and unclear responsibility for mistakes. Aversion falls when people may adjust the algorithm's output even slightly. Later studies also found 'algorithm appreciation' in some tasks, so the direction of the bias depends on domain, framing and expertise.
Managers overrode the demand model after one bad week despite its better record, textbook algorithm aversion.
Origin
Named by Berkeley Dietvorst, Joseph Simmons and Cade Massey in 'Algorithm Aversion: People Erroneously Avoid Algorithms after Seeing Them Err' (Journal of Experimental Psychology: General, 2015), extending the clinical-versus-statistical prediction debate opened by Paul Meehl in 1954.
Related
- automation biasautomation bias, n.: The habit of trusting a computer's suggestion too much: following it even when it is wrong, or missing problems because the machine did not flag them.
- overrelianceoverreliance, n.: Leaning on an AI system more than its reliability justifies: accepting its answers without checking, or letting it make decisions it often gets wrong.
- human-in-the-loophuman-in-the-loop, adj.: Describes AI systems in which a person checks, corrects or approves the machine's work at key points, rather than letting it act entirely on its own.
- trustworthy AItrustworthy AI, n.: AI that deserves people's trust because it works reliably, obeys the law, treats people fairly and can be checked, rather than AI that people merely happen to trust.
- centaurcentaur, n.: A human and an AI working as a team, each doing what it does best, like a chess player who consults a computer before every move. The name comes from the mythical half-human, half-horse creature.
- automation biasautomation bias, n.: The habit of trusting a computer's suggestion too much: following it even when it is wrong, or missing problems because the machine did not flag them.
Sources
- Dietvorst, B. J., Simmons, J. P. & Massey, C. (2015). Algorithm Aversion: People Erroneously Avoid Algorithms after Seeing Them Err. Journal of Experimental Psychology: General 144(1), 114–126.
- Logg, J. M., Minson, J. A. & Moore, D. A. (2019). Algorithm appreciation: People prefer algorithmic to human judgment. Organizational Behavior and Human Decision Processes 151, 90–103.
Mentioned in
algorithmalgorithm, n.: A precise, step-by-step procedure for solving a problem or doing a task, like a recipe a computer can follow. In the news, 'the algorithm' often means software that ranks what you see online., automation biasautomation bias, n.: The habit of trusting a computer's suggestion too much: following it even when it is wrong, or missing problems because the machine did not flag them., Daniel KahnemanDaniel Kahneman, prop. n.: An Israeli-American psychologist who showed how human judgment systematically errs and popularized the idea of fast, intuitive versus slow, deliberate thinking, now used to describe AI reasoning., overrelianceoverreliance, n.: Leaning on an AI system more than its reliability justifies: accepting its answers without checking, or letting it make decisions it often gets wrong.