trolley problem
In plain wordsA thought experiment asking whether you should divert a runaway tram so that it kills one person instead of five, often used to ask how self-driving cars should choose in an unavoidable crash.
A family of moral thought experiments in which an agent can redirect an unavoidable harm, classically a runaway trolley about to kill five people that can be switched onto a side track where it will kill one. Variants, such as stopping the trolley by pushing a person off a footbridge, probe whether what matters morally is consequences, the difference between killing and letting die, or using a person as a means. In AI the problem became a popular frame for programming the crash behavior of autonomous vehiclesautonomous vehicle, n.: A vehicle, usually a car, truck or shuttle, that can drive itself, sensing the road, deciding what to do and steering, braking and accelerating with no human driving., notably in the Moral Machine experiment, whose online survey collected some 40 million choices from respondents in 233 countries and territories and found both widely shared and culturally varying preferences. Many engineers and ethicists criticize the framing: real crashes rarely present clean binary choices, and the substantive questions concern how everyday driving policy distributes risk, who bears liability and what level of risk is acceptable.
The engineers said the trolley problem was the wrong question; how much room to leave cyclists was the right one.
Origin
Posed by philosopher Philippa Foot in 'The Problem of Abortion and the Doctrine of the Double Effect' (1967); Judith Jarvis Thomson named it the trolley problem in 1976 and developed variants such as the footbridge case in that and later work, notably 'The Trolley Problem' (1985). Jean-François Bonnefon, Azim Shariff and Iyad Rahwan applied it to autonomous vehicles in Science (2016), leading to the Moral Machine experiment led by Edmond Awad (2018).
Related
- machine ethicsmachine ethics, n.: The effort to build AI systems that can tell right from wrong and act ethically themselves, rather than only being used ethically by people.
- AI ethicsAI ethics, n.: The study of what is right and wrong in how AI is built and used: who it may harm, who is responsible when it fails, and which values it should respect.
- autonomous vehicleautonomous vehicle, n.: A vehicle, usually a car, truck or shuttle, that can drive itself, sensing the road, deciding what to do and steering, braking and accelerating with no human driving.
- responsibility gapresponsibility gap, n.: The problem that when an AI system that learns or acts on its own causes harm, it may be unclear whether anyone (its makers, users or operators) can fairly be held responsible.
- value alignmentvalue alignment, n.: Building AI whose goals reflect human values — what people care about and consider right — rather than one that merely follows orders or chases a narrow target.
Sources
- Foot, P. (1967). The Problem of Abortion and the Doctrine of the Double Effect. Oxford Review 5, 5–15.
- Thomson, J. J. (1985). The Trolley Problem. Yale Law Journal 94(6), 1395–1415.
- Awad, E. et al. (2018). The Moral Machine experiment. Nature 563, 59–64.