iterative nullspace projection
In plain wordsAn early method for removing a trait such as gender from a model's internal representations by repeatedly training a simple detector for it and then flattening away whatever the detector relied on.
A method for linear concept erasureconcept erasure, n.: Removing a specific idea — such as a person's gender, or an artist's style — from what an AI model can represent or produce, so the model can no longer use or show it. that trains a linear classifier to predict a protected attribute (such as gender) from representations, projects the representations onto the classifier's null space — the subspace the classifier cannot see — and repeats until no linear classifier performs above chance. Composing the projections removes the attribute's linearly decodable information while leaving orthogonal directions untouched. It was used to debias word embeddingsword embedding, n.: A list of numbers standing for a word, learned so that words used in similar ways get similar lists. It lets a computer measure that 'cat' is closer to 'dog' than to 'car'. and to make classifiers built on contextual representations, such as profession prediction from biographies, less dependent on gender. It also underlies amnesic probing, which tests whether a model uses a property by erasing it and measuring the change in behavior. Later methods such as R-LACE and LEACELEACE, abbr.: A one-step mathematical method for scrubbing a concept, such as gender, from an AI model's internal data so that no simple detector can find it, while changing everything else as little as possible. erase with fewer dimensions removed and less collateral damage.
INLP needed 35 iterations before the gender classifier fell to chance accuracy.
Origin
Introduced by Shauli Ravfogel, Yanai Elazar, Hila Gonen, Michael Twiton and Yoav Goldberg in 'Null It Out: Guarding Protected Attributes by Iterative Nullspace Projection' (ACL 2020).
Usage note
Each iteration removes a whole direction, so INLP can strip more information than necessary, and like all linear methods it guards only against linear classifiers.
Related
- concept erasureconcept erasure, n.: Removing a specific idea — such as a person's gender, or an artist's style — from what an AI model can represent or produce, so the model can no longer use or show it.
- LEACELEACE, abbr.: A one-step mathematical method for scrubbing a concept, such as gender, from an AI model's internal data so that no simple detector can find it, while changing everything else as little as possible.
- directional ablationdirectional ablation, n.: Erasing one specific direction from an AI model's internal signals while it runs, so whatever that direction encodes — such as the tendency to refuse — can no longer influence what the model does.
- protected attributeprotected attribute, n.: A personal trait, such as race, sex, religion, age or disability, that the law says must not be used to treat people unfairly.
- word embeddingword embedding, n.: A list of numbers standing for a word, learned so that words used in similar ways get similar lists. It lets a computer measure that 'cat' is closer to 'dog' than to 'car'.
- probingprobing, n.: Testing what information is hidden inside an AI model by training a small, separate predictor on the model's internal activity: if the predictor can recover something, the model has stored it somewhere.
Sources
- Ravfogel, S., Elazar, Y., Gonen, H., Twiton, M. & Goldberg, Y. (2020). Null It Out: Guarding Protected Attributes by Iterative Nullspace Projection. Proceedings of ACL 2020.
Mentioned in
concept erasureconcept erasure, n.: Removing a specific idea — such as a person's gender, or an artist's style — from what an AI model can represent or produce, so the model can no longer use or show it., directional ablationdirectional ablation, n.: Erasing one specific direction from an AI model's internal signals while it runs, so whatever that direction encodes — such as the tendency to refuse — can no longer influence what the model does., LEACELEACE, abbr.: A one-step mathematical method for scrubbing a concept, such as gender, from an AI model's internal data so that no simple detector can find it, while changing everything else as little as possible.