suffering risk
In plain wordsA risk that the future contains suffering on a vast scale, far beyond anything in history, for example if powerful AI created enormous numbers of suffering beings.
A risk that the future will hold suffering on a scale dwarfing everything in Earth's history, or in the phrase of the researchers who defined the category, 'suffering on an astronomical scale'. S-risks overlap with existential risksexistential risk, n.: A danger that could end humanity, or shut it out of a good future forever. Some experts count advanced AI among the largest such risks; others think this worry is overblown. but are distinct: a future could be full of suffering without extinction, and for those who weigh suffering heavily such futures are worse than extinction. Scenarios discussed include misaligned or malevolent AI that creates or simulates vast numbers of sentient beings in distress (what Nick Bostrom calls mind crime), conflict between AI systems involving threats against what the other side values, the spread of practices like factory farming into space, and the lock-in of callous values. Research directions include cooperative AI, surrogate goals that deflect threats, and preventing AI systems from acquiring sadistic or spiteful dispositions; the questions overlap with AI welfareAI welfare, n.: Concern for the well-being of AI systems themselves, in case some of them can suffer or have interests that matter — and the research into how to tell and what to do about it. and the moral patienthoodmoral patienthood, n.: Being the kind of being whose interests matter morally for its own sake, so that it can be wronged. Babies and animals count, even though they are not responsible for what they do. of digital mindsdigital mind, n.: A mind that exists as software running on computers — such as a sufficiently advanced AI or a digital copy of a human brain — especially when considering its moral and social standing..
For an s-risk researcher, a future of trillions of suffering simulations is worse than no future at all.
Origin
Emerged in 2016–2017 at the Foundational Research Institute (now the Center on Long-Term Risk): David Althaus and Lukas Gloor's 2016 report 'Reducing Risks of Astronomical Suffering: A Neglected Priority' built on Brian Tomasik's essays on astronomical suffering, and the short form s-risk spread in effective-altruist circles through Max Daniel's 2017 talk 'S-risks: Why they are the worst existential risks, and how to prevent them' and Tobias Baumann's writing. Earliest use not established.
Related
- existential riskexistential risk, n.: A danger that could end humanity, or shut it out of a good future forever. Some experts count advanced AI among the largest such risks; others think this worry is overblown.
- AI welfareAI welfare, n.: Concern for the well-being of AI systems themselves, in case some of them can suffer or have interests that matter — and the research into how to tell and what to do about it.
- moral patienthoodmoral patienthood, n.: Being the kind of being whose interests matter morally for its own sake, so that it can be wronged. Babies and animals count, even though they are not responsible for what they do.
- value lock-invalue lock-in, n.: The possibility that powerful AI could freeze one set of values or one power structure in place for an extremely long time, so that later generations could never change course.
- longtermismlongtermism, n.: The belief that we should care a great deal about how things go for humanity thousands or millions of years from now, since far more people may live then than have ever lived.
- digital minddigital mind, n.: A mind that exists as software running on computers — such as a sufficiently advanced AI or a digital copy of a human brain — especially when considering its moral and social standing.
Mentioned in
AI welfareAI welfare, n.: Concern for the well-being of AI systems themselves, in case some of them can suffer or have interests that matter — and the research into how to tell and what to do about it., existential riskexistential risk, n.: A danger that could end humanity, or shut it out of a good future forever. Some experts count advanced AI among the largest such risks; others think this worry is overblown., longtermismlongtermism, n.: The belief that we should care a great deal about how things go for humanity thousands or millions of years from now, since far more people may live then than have ever lived., value lock-invalue lock-in, n.: The possibility that powerful AI could freeze one set of values or one power structure in place for an extremely long time, so that later generations could never change course.