gradient inversion attack
In plain wordsRebuilding someone's private training data — such as photos or text — just from the model updates their device shared during collaborative training.
A privacy attack that reconstructs training examples from the gradientsgradient, n.: A list of slopes showing which way, and how steeply, a quantity rises as each of its inputs changes. Training nudges an AI's settings in the opposite direction, downhill. or weight updates a client shares, as in federated learningfederated learning, n.: Training one shared AI model across many phones, hospitals or companies without collecting their data in one place: each trains on its own data and sends back only model updates.. The attacker initializes dummy inputs and labels and optimizes them so that the gradient they would produce matches the observed one; with a small batch the solution often recovers recognizable images or exact text. Labels can sometimes be read off directly from the final layer's gradient, and image priors or a malicious server that modifies the model can make reconstruction work for larger batches. The attack shows that sharing updates rather than data is not a privacy guarantee. Defenses include secure aggregationsecure aggregation, n.: A cryptographic trick that lets a server add up updates from many devices while learning only the total, never any single device's contribution., differential privacydifferential privacy, n.: A mathematical promise that the results of analyzing a dataset — or a model trained on it — would look almost the same whether or not any one person's data was included, so little can be learned about that person. noise, larger batches and gradient compression, though only differential privacy gives provable protection.
From a single shared gradient, the gradient inversion attack recovered a near-pixel-perfect copy of the patient's scan.
Origin
Introduced by Ligeng Zhu, Zhijian Liu and Song Han in 'Deep Leakage from Gradients' (NeurIPS 2019). Bo Zhao and colleagues (iDLG, 2020) extracted labels analytically, and Jonas Geiping and colleagues' 'Inverting Gradients — How easy is it to break privacy in federated learning?' (NeurIPS 2020) extended reconstruction to realistic networks and batches.
Related
- federated learningfederated learning, n.: Training one shared AI model across many phones, hospitals or companies without collecting their data in one place: each trains on its own data and sends back only model updates.
- model inversionmodel inversion, n.: Using a trained AI model's answers to work backwards to private information it learned from — for example, rebuilding a recognizable picture of someone's face from a face-recognition system.
- secure aggregationsecure aggregation, n.: A cryptographic trick that lets a server add up updates from many devices while learning only the total, never any single device's contribution.
- differential privacydifferential privacy, n.: A mathematical promise that the results of analyzing a dataset — or a model trained on it — would look almost the same whether or not any one person's data was included, so little can be learned about that person.
- membership inference attackmembership inference attack, n.: An attack that tries to tell whether a particular person's record, or a particular document, was part of an AI model's training data — which can itself reveal private facts.
Sources
- Zhu, L., Liu, Z. & Han, S. (2019). Deep Leakage from Gradients. NeurIPS 2019.
Mentioned in
federated learningfederated learning, n.: Training one shared AI model across many phones, hospitals or companies without collecting their data in one place: each trains on its own data and sends back only model updates., model inversionmodel inversion, n.: Using a trained AI model's answers to work backwards to private information it learned from — for example, rebuilding a recognizable picture of someone's face from a face-recognition system., secure aggregationsecure aggregation, n.: A cryptographic trick that lets a server add up updates from many devices while learning only the total, never any single device's contribution.