Skip to content
Last Dictionary
Plane
Security & MisuseTraining & Post-trainingMathematics & Statistics

DP-SGD

abbreviationAlso: differentially private stochastic gradient descent, differentially private SGD, DP training

In plain wordsA way of training a neural network so that it cannot learn too much about any single person in the data: each example's influence is capped, and random noise is added at every step.

  1. Differentially private stochastic gradient descentstochastic gradient descent, n.: A faster version of gradient descent that estimates which way is downhill from a small random sample of the training data at each step, instead of checking the whole dataset.: a training algorithm that gives a model a differential privacydifferential privacy, n.: A mathematical promise that the results of analyzing a dataset — or a model trained on it — would look almost the same whether or not any one person's data was included, so little can be learned about that person. guarantee with respect to its training examples. At each step it computes per-example gradientsgradient, n.: A list of slopes showing which way, and how steeply, a quantity rises as each of its inputs changes. Training nudges an AI's settings in the opposite direction, downhill., clips each to a maximum ℓ2 norm C so that no single example can move the model by more than a bounded amount, sums them, adds Gaussian noise with standard deviation proportional to C, and averages. A privacy accountant (originally the moments accountant, now usually Rényi or numerical accountants) tracks the cumulative ε across steps, exploiting the privacy amplification provided by random batch sampling. DP-SGD costs accuracy, needs large batches and more computation, and is most often used to fine-tune pretrained models on sensitive data; libraries such as Opacus and TensorFlow Privacy implement it.

    Fine-tuning with DP-SGD at ε = 8 cost two points of accuracy relative to the non-private model.

g~=1B(∑i=1Bgimax⁡(1,∥gi∥2/C)+N(0,σ2C2I))\tilde{g} = \frac{1}{B}\Big(\sum_{i=1}^{B} \frac{g_i}{\max\big(1, \|g_i\|_2 / C\big)} + \mathcal{N}\big(0, \sigma^2 C^2 \mathbf{I}\big)\Big)

Origin

Noisy private gradient descent was studied by Shuang Song, Kamalika Chaudhuri and Anand Sarwate (2013) and by Raef Bassily, Adam Smith and Abhradeep Thakurta (2014). Martín Abadi and colleagues at Google made it practical for deep networks in 'Deep Learning with Differential Privacy' (ACM CCS 2016), introducing the moments accountant; the name DP-SGD spread from that paper.

Related

See also
  • differential privacydifferential privacy, n.: A mathematical promise that the results of analyzing a dataset — or a model trained on it — would look almost the same whether or not any one person's data was included, so little can be learned about that person.
  • privacy budgetprivacy budget, n.: The total amount of privacy risk a data holder agrees to spend; each analysis or release uses up part of it, and once it is gone, no more answers can be given safely.
  • stochastic gradient descentstochastic gradient descent, n.: A faster version of gradient descent that estimates which way is downhill from a small random sample of the training data at each step, instead of checking the whole dataset.
  • PATEPATE, abbr.: A privacy method in which several 'teacher' models, each trained on a separate slice of sensitive data, vote noisily to label public examples, and a 'student' model learns only from those noisy votes.
  • memorizationmemorization, n.: When an AI model stores specific pieces of its training data — a paragraph, a phone number, an image — closely enough to reproduce them, instead of only learning general patterns.
  • membership inference attackmembership inference attack, n.: An attack that tries to tell whether a particular person's record, or a particular document, was part of an AI model's training data — which can itself reveal private facts.

Sources

  1. Abadi, M., Chu, A., Goodfellow, I., McMahan, H. B., Mironov, I., Talwar, K. & Zhang, L. (2016). Deep Learning with Differential Privacy. ACM CCS 2016. arXiv:1607.00133.

Mentioned in

differential privacydifferential privacy, n.: A mathematical promise that the results of analyzing a dataset — or a model trained on it — would look almost the same whether or not any one person's data was included, so little can be learned about that person., gradient clippinggradient clipping, n.: A safety limit on how big a single training update can be. If the computed adjustment is unusually large, it is scaled down so that one bad batch cannot wreck the model., privacy-preserving machine learningprivacy-preserving machine learning, n.: The family of techniques for training and using AI models while keeping the underlying personal or confidential data hidden — from adding careful noise to computing on encrypted data., stochastic gradient descentstochastic gradient descent, n.: A faster version of gradient descent that estimates which way is downhill from a small random sample of the training data at each step, instead of checking the whole dataset., Cynthia DworkCynthia Dwork, prop. n.: An American computer scientist who co-invented differential privacy, the leading mathematical definition of privacy in data analysis, and helped found the study of fairness in algorithms., PATEPATE, abbr.: A privacy method in which several 'teacher' models, each trained on a separate slice of sensitive data, vote noisily to label public examples, and a 'student' model learns only from those noisy votes., privacy budgetprivacy budget, n.: The total amount of privacy risk a data holder agrees to spend; each analysis or release uses up part of it, and once it is gone, no more answers can be given safely.