model inversion
In plain wordsUsing a trained AI model's answers to work backwards to private information it learned from — for example, rebuilding a recognizable picture of someone's face from a face-recognition system.
A privacy attack that uses a trained model's outputs, and sometimes its parameters, to reconstruct sensitive features of its training datatraining data, n.: The examples an AI system learns from. Whatever patterns, skills, gaps and biases are in the training data tend to show up in the finished system. or of the classes it recognizes. Given a label or a partial record, the attacker searches for inputs that the model scores as most likely for that label, often constrained by a generative prior so that results look realistic: recovering a representative face for a name in a face-recognition model, for instance, or inferring a sensitive attribute such as a genetic marker from a model's prediction and a person's other attributes. Inversion typically reveals class-representative information rather than exact records, which distinguishes it from training data extractiontraining data extraction, n.: Getting an AI model to repeat, word for word, pieces of the data it was trained on, which may include someone's private contact details or copyrighted text. and membership inferencemembership inference attack, n.: An attack that tries to tell whether a particular person's record, or a particular document, was part of an AI model's training data — which can itself reveal private facts.. Differential privacydifferential privacy, n.: A mathematical promise that the results of analyzing a dataset — or a model trained on it — would look almost the same whether or not any one person's data was included, so little can be learned about that person., coarser outputs and withholding confidence scores are standard mitigations.
Using only the classifier's confidence scores, model inversion produced recognizable images of three enrolled faces.
Origin
Matt Fredrikson and colleagues introduced model inversion in 2014, inferring patients' genetic markers from a warfarin-dosing model (USENIX Security 2014); Fredrikson, Somesh Jha and Thomas Ristenpart extended it to face recognition using confidence scores (ACM CCS 2015). Attacks using generative priors (Yuheng Zhang and colleagues, 2020) and gradient inversiongradient inversion attack, n.: Rebuilding someone's private training data — such as photos or text — just from the model updates their device shared during collaborative training. in federated learning followed.
Related
- membership inference attackmembership inference attack, n.: An attack that tries to tell whether a particular person's record, or a particular document, was part of an AI model's training data — which can itself reveal private facts.
- training data extractiontraining data extraction, n.: Getting an AI model to repeat, word for word, pieces of the data it was trained on, which may include someone's private contact details or copyrighted text.
- gradient inversion attackgradient inversion attack, n.: Rebuilding someone's private training data — such as photos or text — just from the model updates their device shared during collaborative training.
- differential privacydifferential privacy, n.: A mathematical promise that the results of analyzing a dataset — or a model trained on it — would look almost the same whether or not any one person's data was included, so little can be learned about that person.
- memorizationmemorization, n.: When an AI model stores specific pieces of its training data — a paragraph, a phone number, an image — closely enough to reproduce them, instead of only learning general patterns.
Sources
- Fredrikson, M., Jha, S. & Ristenpart, T. (2015). Model Inversion Attacks that Exploit Confidence Information and Basic Countermeasures. ACM CCS 2015.
Mentioned in
adversarial machine learningadversarial machine learning, n.: The study of how machine-learning systems can be fooled, corrupted or spied on by someone working deliberately against them, and of how to build systems that hold up., membership inference attackmembership inference attack, n.: An attack that tries to tell whether a particular person's record, or a particular document, was part of an AI model's training data — which can itself reveal private facts., training data extractiontraining data extraction, n.: Getting an AI model to repeat, word for word, pieces of the data it was trained on, which may include someone's private contact details or copyrighted text., gradient inversion attackgradient inversion attack, n.: Rebuilding someone's private training data — such as photos or text — just from the model updates their device shared during collaborative training., textual inversiontextual inversion, n.: Teaching an image generator a new 'word' for a specific object or style from a few example pictures, leaving the model itself unchanged — only one vocabulary entry is added.