factored cognition

The hypothesis, and the research program testing it, that complex cognitive work, such as answering hard questions or writing and evaluating arguments, can be split into many small pieces, each handled in a short time by someone who never sees the whole problem, and each checkable on its own. It is the key assumption behind iterated amplificationiterated amplification, n.: A proposed training scheme in which a person, helped by many copies of an AI, solves harder problems than either could alone; a new AI then learns to imitate that team, and the cycle repeats., HCHHCH, abbr.: A thought experiment: a person who can hand parts of a problem to copies of themselves, who can hand parts on again, endlessly — the ideal some AI training schemes try to approximate. and some forms of AI debateAI debate, n.: A proposed way to supervise very capable AI: two AI systems argue opposite answers to a question and a human judge decides who argued better, on the theory that lies are easier to expose than to defend.: if it holds, a human overseer never needs to understand an entire superhuman computation, only individual steps. Ought's experiments of this kind, in which participants with a few minutes and limited context answered decomposed subquestions or evaluated arguments one step at a time, did not settle the question. Its practical descendants include task decompositiontask decomposition, n.: Breaking a big, vague job into smaller, concrete steps that can each be done and checked — the way a recipe splits a meal into stages. and orchestrator–workerorchestrator–worker pattern, n.: A way of organizing AI teamwork, like a newsroom: an editor AI decides which stories a big assignment needs, sends reporter AIs to cover them at the same time, and turns what they file into one report. agent designs, though these serve capability rather than oversight.

If factored cognition fails for research taste, amplification cannot supervise the hardest parts of science.