data attribution
In plain wordsWorking out which training examples were responsible for a particular behavior or answer of an AI model — useful for explaining, debugging and crediting the data behind it.
- 1
Machine learning
The estimation of how individual training examples, or groups of them, influenced a trained model's parameters, predictions or behaviors. The ideal is counterfactual — how would an output change if an example had been left out or upweighted? — but retraining for every example is infeasible, so methods approximate it. Influence functions, from robust statistics, use gradients and an approximation of the inverse HessianHessian, n.: A table of 'slopes of slopes' that describes how a surface curves: whether a point is the bottom of a bowl, the top of a hill or a mountain pass.; TracIn sums gradient similarities along the training trajectory; datamodels and TRAK fit simple predictors of outputs from training-subset membership; Data Shapley assigns credit with the Shapley valueShapley value, n.: A rule from game theory for splitting a shared reward fairly among team members, by averaging how much each member adds when joining the team in every possible order.. Uses include explaining outputs, finding mislabeled or harmful data, detecting data poisoningdata poisoning, n.: Sneaking bad examples into the data an AI learns from, so the finished model behaves as the attacker wants — badly overall, or wrongly only in special situations. and memorizationmemorization, n.: When an AI model stores specific pieces of its training data — a paragraph, a phone number, an image — closely enough to reproduce them, instead of only learning general patterns., selecting training data, and studying how large modelslarge language model, n.: A computer program trained on enormous amounts of text to predict the words that come next. At vast scale, that simple skill lets it chat, answer questions, translate, summarize and write code. generalize. The approximations become fragile for large non-convex models, and different methods often disagree.
Data attribution traced the odd answer to a handful of near-duplicate forum posts.
- 2
Data rights
The assignment of credit, and potentially payment, to the people or organizations whose data contributed to a model or to a particular output, as demanded by some licenses, creators' groups and proposed compensation schemes; technical attribution methods are sometimes proposed as its basis, though they cannot yet bear that weight at scale.
Origin
Influence functions come from Frank Hampel's robust statistics (1974) and Dennis Cook's regression diagnostics (1977). Pang Wei Koh and Percy Liang's 'Understanding Black-box Predictions via Influence Functions' (ICML 2017 best paper) brought them to deep learning, and Anthropic researchers scaled them to large language models in 2023.
Usage note
Distinct from feature attributionfeature attribution, n.: Giving each input to an AI model — each pixel, word or data column — a score for how much it pushed a particular output up or down., which explains an output in terms of parts of the input (pixels, words) rather than training examples. Attribution scores estimate statistical influence, not legal or moral responsibility.
Related
- Shapley valueShapley value, n.: A rule from game theory for splitting a shared reward fairly among team members, by averaging how much each member adds when joining the team in every possible order.
- interpretabilityinterpretability, n.: The effort to understand how an AI system arrives at its outputs — what information it uses and how it combines it — instead of treating it as a sealed black box.
- data provenancedata provenance, n.: The documented history of a piece of data: where it came from, who created and changed it, under what permissions, and how it was processed before reaching an AI system.
- memorizationmemorization, n.: When an AI model stores specific pieces of its training data — a paragraph, a phone number, an image — closely enough to reproduce them, instead of only learning general patterns.
- data poisoningdata poisoning, n.: Sneaking bad examples into the data an AI learns from, so the finished model behaves as the attacker wants — badly overall, or wrongly only in special situations.
- machine unlearningmachine unlearning, n.: Erasing specific things a trained AI model has learned, such as dangerous lab techniques, a copyrighted novel or one person's records, far more cheaply than training the model all over again.
- feature attributionfeature attribution, n.: Giving each input to an AI model — each pixel, word or data column — a score for how much it pushed a particular output up or down.
Sources
- Koh, P. W. & Liang, P. (2017). Understanding Black-box Predictions via Influence Functions. ICML 2017.
- Grosse, R. et al. (2023). Studying Large Language Model Generalization with Influence Functions. arXiv preprint.
- Park, S. M. et al. (2023). TRAK: Attributing Model Behavior at Scale. ICML 2023.
Mentioned in
data provenancedata provenance, n.: The documented history of a piece of data: where it came from, who created and changed it, under what permissions, and how it was processed before reaching an AI system., feature attributionfeature attribution, n.: Giving each input to an AI model — each pixel, word or data column — a score for how much it pushed a particular output up or down., Shapley valueShapley value, n.: A rule from game theory for splitting a shared reward fairly among team members, by averaging how much each member adds when joining the team in every possible order., influence functioninfluence function, n.: A mathematical shortcut for estimating how much a single training example shaped a model's behavior, for instance how a prediction would change if that example had been left out, without retraining the model.