rationalist community
In plain wordsA loose online community that prizes careful reasoning and spotting one's own biases; gathered around the forum LessWrong, it was one of the first groups to take the dangers of advanced AI seriously.
A loose network of writers, researchers and readers, largely online and concentrated in the San Francisco Bay Area, organized around improving human reasoning — Bayesian probability, calibrated forecasting, awareness of cognitive bias, decision theory — and around the belief that advanced AI could pose an existential riskexistential risk, n.: A danger that could end humanity, or shut it out of a good future forever. Some experts count advanced AI among the largest such risks; others think this worry is overblown.. Its canonical texts include Eliezer Yudkowsky's Sequences, LessWrongLessWrong, prop. n.: An online forum, founded in 2009, about thinking clearly and avoiding mental biases; it became a main gathering place for people worried about the dangers of advanced AI. and Scott Alexander's blogs Slate Star Codex and Astral Codex Ten. The community overlaps heavily with effective altruismeffective altruism, n.: A movement that treats charity like a research problem, comparing causes by how much good each dollar or working hour achieves; many of its members came to rank risks from advanced AI as a top priority. and supplied many of the founders, researchers and funders of early AI safetyAI safety, n.: The effort to keep AI from doing serious harm, whether because it pursues aims its makers never intended, is turned to bad uses or simply malfunctions — a concern that grows as systems become more capable. organizations, along with characteristic vocabulary such as 'updating', 'steelmanning', 'priors' and p(doom)p(doom), n.: Shorthand for someone's personal estimate of the chance that AI leads to catastrophe for humanity, such as extinction. People give it as a percentage, from nearly zero to nearly certain.. Critics describe it as insular and point to harmful splinter groups; people sympathetic to it but not part of it are called 'rationalist-adjacent'.
Half the alignment team came up through the rationalist community, and it shows in how they argue.
Origin
Grew from the blogs Overcoming Bias (2006) and LessWrongLessWrong, prop. n.: An online forum, founded in 2009, about thinking clearly and avoiding mental biases; it became a main gathering place for people worried about the dangers of advanced AI. (2009), whose members had adopted 'rationalist' as a self-description by about 2010. The Center for Applied Rationality (founded 2012) ran workshops, and Slate Star Codex (2013–2020) widened the readership. The AI boom of the 2020s brought the community's ideas, and its disputes with accelerationistseffective accelerationism, n.: An online movement, known as e/acc, that wants AI and other technology developed as fast as possible, arguing that progress is good and that safety-driven regulation does more harm than good., into mainstream coverage.
Usage note
Not to be confused with rationalism in philosophy (Descartes, Spinoza, Leibniz), the view that reason rather than sense experience is the chief source of knowledge.
Related
- LessWrongLessWrong, prop. n.: An online forum, founded in 2009, about thinking clearly and avoiding mental biases; it became a main gathering place for people worried about the dangers of advanced AI.
- effective altruismeffective altruism, n.: A movement that treats charity like a research problem, comparing causes by how much good each dollar or working hour achieves; many of its members came to rank risks from advanced AI as a top priority.
- Eliezer YudkowskyEliezer Yudkowsky, prop. n.: An American writer and AI theorist who, from around 2000, argued that superhuman AI would likely destroy humanity unless its goals were made safe, and who founded an online community devoted to rational thinking.
- TESCREALTESCREAL, abbr.: A critical label for a cluster of related belief systems, including transhumanism, effective altruism and longtermism, that its coiners say drive the push to build AGI and share roots with eugenics.
- doomerdoomer, n.: Someone who believes advanced AI is likely to cause a catastrophe, possibly human extinction; often used as a put-down by people who think those fears are exaggerated.
- p(doom)p(doom), n.: Shorthand for someone's personal estimate of the chance that AI leads to catastrophe for humanity, such as extinction. People give it as a percentage, from nearly zero to nearly certain.
- MolochMoloch, prop. n.: A name, borrowed from an ancient god of sacrifice, for situations where competition forces everyone to act against their shared interests — as when AI companies feel pushed to race ahead despite the risks.
Mentioned in
effective altruismeffective altruism, n.: A movement that treats charity like a research problem, comparing causes by how much good each dollar or working hour achieves; many of its members came to rank risks from advanced AI as a top priority., Eliezer YudkowskyEliezer Yudkowsky, prop. n.: An American writer and AI theorist who, from around 2000, argued that superhuman AI would likely destroy humanity unless its goals were made safe, and who founded an online community devoted to rational thinking., LessWrongLessWrong, prop. n.: An online forum, founded in 2009, about thinking clearly and avoiding mental biases; it became a main gathering place for people worried about the dangers of advanced AI., Machine Intelligence Research InstituteMachine Intelligence Research Institute, prop. n.: A small nonprofit in Berkeley, California, among the first to warn that smarter-than-human AI could destroy humanity, which now campaigns for a worldwide halt to building it., infohazardinfohazard, n.: Information that could cause harm simply by being shared or known — for example, details that would make a dangerous weapon easier to build., MolochMoloch, prop. n.: A name, borrowed from an ancient god of sacrifice, for situations where competition forces everyone to act against their shared interests — as when AI companies feel pushed to race ahead despite the risks., Roko's basiliskRoko's basilisk, n.: A notorious internet thought experiment: a future super-powerful AI might punish everyone who foresaw it but did not help bring it about, so merely hearing the argument would put you at risk. Almost nobody accepts it., TESCREALTESCREAL, abbr.: A critical label for a cluster of related belief systems, including transhumanism, effective altruism and longtermism, that its coiners say drive the push to build AGI and share roots with eugenics., transhumanismtranshumanism, n.: A movement holding that people should use technology, such as genetic engineering, brain implants or AI, to overcome human limits like aging, disease and death, and become more than human., updateless decision theoryupdateless decision theory, n.: A theory of rational choice in which an agent commits to the best overall plan for every situation it might face, instead of re-deciding each time it learns something new.