Chinese room
In plain wordsA thought experiment: a person who knows no Chinese follows a rulebook to answer Chinese messages so well that outsiders think they understand. If the person doesn't understand, does a computer running a program?
A thought experiment and argument by John Searle against strong AIstrong AI, n.: Originally, the claim that a computer running the right program would literally have a mind and understand, not just imitate understanding. Today it often simply means AI as capable as humans across the board.. In the scenario Searle, who reads no Chinese, sits in a closed room with a rulebook written in English. Notes in Chinese characters come in, the rulebook tells him purely by the characters' shapes which ones to send back, and his replies are so apt that people outside take the room to hold a fluent speaker. Yet he grasps none of it, and he is doing exactly what a computer executing a program does; so, Searle concludes, running a program can never by itself produce understanding: formal symbol manipulation (syntax) is not enough for meaning (semantics). Leading replies include the systems reply (the whole room, not the man, understands), the robot reply (connect the program to sensors and motors), the brain simulator reply and the other minds reply. The argument is regularly revived in debates over whether large language modelslarge language model, n.: A computer program trained on enormous amounts of text to predict the words that come next. At vast scale, that simple skill lets it chat, answer questions, translate, summarize and write code. understand what they say.
Whatever you think of the Chinese room, it forces you to say where in the system the understanding is supposed to be.
Origin
Published by John Searle in 'Minds, Brains, and Programs' (Behavioral and Brain Sciences, 1980) with peer commentaries, partly in response to Roger Schank and Robert Abelson's story-understanding programs. Precursors include Gottfried Leibniz's mill (1714) and Ned Block's 'China brain' (1978).
Usage note
The argument targets programs, not machines: Searle grants that an artificial system with the brain's causal powers could think. Whether it applies to neural networksneural network, n.: A kind of computer program that learns a task from examples by tuning thousands to billions of connections between simple number-crunching units, an arrangement loosely inspired by nerve cells in the brain., whose operations are not rule-following in the rulebook sense, is itself disputed.
Related
- strong AIstrong AI, n.: Originally, the claim that a computer running the right program would literally have a mind and understand, not just imitate understanding. Today it often simply means AI as capable as humans across the board.
- symbol grounding problemsymbol grounding problem, n.: The puzzle of how words or symbols inside a machine can mean anything by themselves, rather than only pointing to more symbols — like a dictionary that defines every word using other words.
- machine understandingmachine understanding, n.: Whether, and in what sense, an AI system grasps what words, situations or problems mean — rather than just producing answers that look as though it does.
- biological naturalismbiological naturalism, n.: John Searle's view that consciousness is a real biological product of brains, as digestion is of stomachs, so running a computer program would not by itself create a mind.
- intentionalityintentionality, n.: The way thoughts, beliefs and desires are about things: a belief is about rain, a wish is for cake. Philosophers ask whether a machine's internal states can be about anything.
- John SearleJohn Searle, prop. n.: An American philosopher who devised the Chinese room argument, which holds that a computer running a program can manipulate symbols without understanding anything.
- BlockheadBlockhead, n.: An imagined machine that passes any conversation test by looking up prewritten replies in an unimaginably huge table. Most people judge it unintelligent, which suggests that behavior alone does not prove intelligence.
- Turing testTuring test, n.: Alan Turing's proposed test of machine intelligence: a judge holds typed conversations with a hidden person and a hidden computer, and the computer passes if the judge cannot consistently pick it out.
Sources
- Searle, J. R. (1980). Minds, Brains, and Programs. Behavioral and Brain Sciences 3(3), 417–457.
Mentioned in
Alan TuringAlan Turing, prop. n.: A British mathematician who worked out what computers can do in principle, helped break German codes in the Second World War, and proposed the best-known test of machine intelligence., stochastic parrotstochastic parrot, n.: A critical label for AI language systems, likening them to parrots that echo patterns in the text they were trained on, fluently but with no grasp of what the words mean., Turing testTuring test, n.: Alan Turing's proposed test of machine intelligence: a judge holds typed conversations with a hidden person and a hidden computer, and the computer passes if the judge cannot consistently pick it out., computational theory of mindcomputational theory of mind, n.: The idea that thinking is a kind of computation: the mind processes information by following rules, much as a computer runs a program, and the brain is the machine doing it., Computing Machinery and IntelligenceComputing Machinery and Intelligence, prop. n.: Alan Turing's 1950 essay asking whether machines can think. It proposed a conversation test in place of that question, answered objections to thinking machines and suggested that machines might learn like children., functionalismfunctionalism, n.: The view that minds are defined by how they work, not what they are made of: pain, for example, is whatever inner state plays pain's part, linking injury, other thoughts and behavior., John SearleJohn Searle, prop. n.: An American philosopher who devised the Chinese room argument, which holds that a computer running a program can manipulate symbols without understanding anything., machine understandingmachine understanding, n.: Whether, and in what sense, an AI system grasps what words, situations or problems mean — rather than just producing answers that look as though it does., natural language understandingnatural language understanding, n.: The part of language AI concerned with working out what a piece of text means — who did what to whom, what the writer wants and what follows from it — rather than with producing text., philosophical zombiephilosophical zombie, n.: An imagined being that behaves exactly like a person — talking, laughing, even saying it feels pain — but has no inner experience at all. Philosophers use it to ask whether behavior proves a mind., philosophy of artificial intelligencephilosophy of artificial intelligence, n.: The branch of philosophy that asks what AI is and could be: whether machines can think, understand or be conscious, what intelligence is, and what we owe to and should expect from intelligent machines., physical symbol system hypothesisphysical symbol system hypothesis, n.: The 1976 claim by two AI pioneers that a machine which stores and rearranges symbols according to rules has everything needed for general intelligence, and that anything generally intelligent must be such a machine., strong AIstrong AI, n.: Originally, the claim that a computer running the right program would literally have a mind and understand, not just imitate understanding. Today it often simply means AI as capable as humans across the board., symbol grounding problemsymbol grounding problem, n.: The puzzle of how words or symbols inside a machine can mean anything by themselves, rather than only pointing to more symbols — like a dictionary that defines every word using other words., behaviorismbehaviorism, n.: The view that the mind should be studied, or even defined, only through observable behavior rather than inner thoughts or feelings. It shapes debates about whether AI that acts intelligently really is intelligent., biological naturalismbiological naturalism, n.: John Searle's view that consciousness is a real biological product of brains, as digestion is of stomachs, so running a computer program would not by itself create a mind., BlockheadBlockhead, n.: An imagined machine that passes any conversation test by looking up prewritten replies in an unimaginably huge table. Most people judge it unintelligent, which suggests that behavior alone does not prove intelligence., intentionalityintentionality, n.: The way thoughts, beliefs and desires are about things: a belief is about rain, a wish is for cake. Philosophers ask whether a machine's internal states can be about anything., octopus testoctopus test, n.: A thought experiment in which a clever octopus learns to impersonate a stranded islander by eavesdropping on an undersea cable, without ever knowing what the words refer to. It asks whether text-trained AI can grasp meaning.