feedback loop
In plain wordsA cycle in which what a system produces comes back to shape its own future input, like a microphone howling near its loudspeaker. In AI, a model's predictions can shape the very data it later learns from.
- 1
Machine learning systems
A cycle in which a deployed model's outputs influence the data later used to train or evaluate it, or the behavior it is trying to predict. A recommender systemrecommender system, n.: Software that guesses which songs, products, posts or jobs you are most likely to want and shows those first, as when a video app lines up the next clip or a shop suggests an add-on. shows popular items, gathers clicks mostly on what it showed and learns to show those items even more, amplifying popularity bias and narrowing what users see (see filter bubblefilter bubble, n.: A situation in which you see mostly news and opinions you already agree with, because apps choose what to show you based on what you clicked before.); predictive policingpredictive policing, n.: Using computer programs to forecast where crimes are likely to happen, or who is likely to be involved, so that police can decide where to patrol or whom to watch. sends patrols where recorded crime is high, generating more records there; a lender never observes repayment by applicants it rejected, so its data tend to confirm its own decisions. Generative models trained on a web increasingly filled with model output risk model collapsemodel collapse, n.: The gradual decline of AI models trained on the output of earlier AI models: rare details vanish first, then quality and variety erode, like a photocopy of a photocopy.. Such loops can entrench bias, cause distribution shiftdistribution shift, n.: When the data an AI system meets in real use differs from the data it was trained on — new users, places, times or habits — often making it perform worse than tests suggested. and make offline evaluation misleading. Mitigations include exploration (showing some items at random), logging the probability of each decision so that later analysis can correct for it, holdout groups that never see the model, audits and tracking data provenancedata provenance, n.: The documented history of a piece of data: where it came from, who created and changed it, under what permissions, and how it was processed before reaching an AI system.. When predictions change the outcomes they predict, the setting is studied as performative prediction.
Clicks only come from what we already rank highly, so retraining on them set up a feedback loop.
- 2
Cybernetics and control
An arrangement in which a measure of what a system produces is fed back to influence what it does next. Negative (balancing) feedback counteracts deviations from a goal and stabilizes a system, as in a thermostat, a steam-engine governor or the controllercontrol theory, n.: The mathematics of making machines behave as intended by constantly measuring what they are doing and correcting them, as a thermostat, cruise control or self-balancing robot does. of a robot arm; positive (reinforcing) feedback amplifies deviations, producing runaway growth, lock-in or oscillation. Cyberneticscybernetics, n.: The study of how animals, machines and organizations steer themselves using feedback: sensing the result of an action and correcting course. It was both a parent and an early rival of AI. treated negative feedback as the mechanism of purposeful behavior in animals and machines, an idea that shaped early AI and the agent–environment loop of reinforcement learningreinforcement learning, n.: A way of teaching a computer program by trial and error: it tries actions, earns points for good results and penalties for bad ones, and gradually learns which choices pay off over time..
Origin
Engineers analyzed feedback in governors (James Clerk Maxwell, 1868) and amplifiers (Harold Black, 1927). Arturo Rosenblueth, Norbert Wiener and Julian Bigelow linked purpose to negative feedback in 1943, and Wiener made feedback central to cybernetics (1948). In machine learning, D. Sculley and colleagues warned of hidden feedback loops in production systems (2015), and Danielle Ensign and colleagues modeled runaway feedback loops in predictive policing (2018).
Usage note
Positive and negative mean reinforcing and balancing, not good and bad: runaway engagement and bias amplification are positive feedback. Distinct from the human feedback used to train models, as in reinforcement learning from human feedbackreinforcement learning from human feedback, n.: A way of training AI using people's judgments: humans compare pairs of answers, a scoring model learns their preferences, and the AI is then trained to produce answers that score well..
Related
- cyberneticscybernetics, n.: The study of how animals, machines and organizations steer themselves using feedback: sensing the result of an action and correcting course. It was both a parent and an early rival of AI.
- recommender systemrecommender system, n.: Software that guesses which songs, products, posts or jobs you are most likely to want and shows those first, as when a video app lines up the next clip or a shop suggests an add-on.
- model collapsemodel collapse, n.: The gradual decline of AI models trained on the output of earlier AI models: rare details vanish first, then quality and variety erode, like a photocopy of a photocopy.
- predictive policingpredictive policing, n.: Using computer programs to forecast where crimes are likely to happen, or who is likely to be involved, so that police can decide where to patrol or whom to watch.
- filter bubblefilter bubble, n.: A situation in which you see mostly news and opinions you already agree with, because apps choose what to show you based on what you clicked before.
- distribution shiftdistribution shift, n.: When the data an AI system meets in real use differs from the data it was trained on — new users, places, times or habits — often making it perform worse than tests suggested.
- control theorycontrol theory, n.: The mathematics of making machines behave as intended by constantly measuring what they are doing and correcting them, as a thermostat, cruise control or self-balancing robot does.
- Goodhart's lawGoodhart's law, n.: The rule of thumb that once a number is used as a target, people or AIs find ways to push it up that no longer reflect what it was meant to measure.
- reinforcement learning from human feedbackreinforcement learning from human feedback, n.: A way of training AI using people's judgments: humans compare pairs of answers, a scoring model learns their preferences, and the AI is then trained to produce answers that score well.
Sources
- Wiener, N. (1948). Cybernetics: Or Control and Communication in the Animal and the Machine. MIT Press.
- Sculley, D., Holt, G., Golovin, D. et al. (2015). Hidden Technical Debt in Machine Learning Systems. Advances in Neural Information Processing Systems 28 (NeurIPS 2015).
- Ensign, D., Friedler, S. A., Neville, S., Scheidegger, C. & Venkatasubramanian, S. (2018). Runaway Feedback Loops in Predictive Policing. Proceedings of the Conference on Fairness, Accountability and Transparency (FAT* 2018), PMLR 81.