differential privacy
In plain wordsA mathematical promise that the results of analyzing a dataset — or a model trained on it — would look almost the same whether or not any one person's data was included, so little can be learned about that person.
A formal privacy guarantee for randomized algorithms: a mechanism M is (ε, δ)-differentially private if, for every pair of datasets differing in one individual's record and every set of possible outputs, the probability of producing an output in that set changes by at most a factor e^ε plus a small slack δ. Privacy comes from calibrated randomness — for example, adding Laplace or Gaussian noise scaled to a query's sensitivity, the most one record can change the answer. The guarantee holds against attackers with arbitrary side information, survives post-processing and degrades predictably under composition, which allows a privacy budgetprivacy budget, n.: The total amount of privacy risk a data holder agrees to spend; each analysis or release uses up part of it, and once it is gone, no more answers can be given safely. to be tracked across many analyses. In machine learning it is achieved mainly with DP-SGDDP-SGD, abbr.: A way of training a neural network so that it cannot learn too much about any single person in the data: each example's influence is capped, and random noise is added at every step. or PATEPATE, abbr.: A privacy method in which several 'teacher' models, each trained on a separate slice of sensitive data, vote noisily to label public examples, and a 'student' model learns only from those noisy votes., and it provably bounds membership inferencemembership inference attack, n.: An attack that tries to tell whether a particular person's record, or a particular document, was part of an AI model's training data — which can itself reveal private facts. and limits memorizationmemorization, n.: When an AI model stores specific pieces of its training data — a paragraph, a phone number, an image — closely enough to reproduce them, instead of only learning general patterns., at a cost in accuracy. Smaller ε means stronger privacy; values from about 1 to 10 are common in practice.
The hospital released the model only after it was trained with differential privacy at ε = 3.
Origin
Introduced by Cynthia Dwork, Frank McSherry, Kobbi Nissim and Adam Smith in 'Calibrating Noise to Sensitivity in Private Data Analysis' (TCC 2006); the name comes from Dwork's ICALP 2006 paper 'Differential Privacy'. The four shared the 2017 Gödel Prize. Large deployments followed at Google (2014), Apple (2016) and the US Census Bureau, which used it for 2020 census data products.
Usage note
A property of an algorithm, not of a dataset or output: one cannot check a released table and declare it differentially private. Guarantees with large ε (say, above 10) are weak in theory even if attacks remain hard in practice.
Related
- DP-SGDDP-SGD, abbr.: A way of training a neural network so that it cannot learn too much about any single person in the data: each example's influence is capped, and random noise is added at every step.
- privacy budgetprivacy budget, n.: The total amount of privacy risk a data holder agrees to spend; each analysis or release uses up part of it, and once it is gone, no more answers can be given safely.
- local differential privacylocal differential privacy, n.: A privacy method in which each person's device scrambles their data with random noise before sending it anywhere, so the company collecting it never sees anyone's true answer but can still estimate overall trends.
- PATEPATE, abbr.: A privacy method in which several 'teacher' models, each trained on a separate slice of sensitive data, vote noisily to label public examples, and a 'student' model learns only from those noisy votes.
- membership inference attackmembership inference attack, n.: An attack that tries to tell whether a particular person's record, or a particular document, was part of an AI model's training data — which can itself reveal private facts.
- privacy-preserving machine learningprivacy-preserving machine learning, n.: The family of techniques for training and using AI models while keeping the underlying personal or confidential data hidden — from adding careful noise to computing on encrypted data.
- federated learningfederated learning, n.: Training one shared AI model across many phones, hospitals or companies without collecting their data in one place: each trains on its own data and sends back only model updates.
- k-anonymityk-anonymity, n.: A rule for anonymizing tables of personal data: blur details such as age and postcode until every person matches at least k others, so no one can be singled out.
Sources
- Dwork, C., McSherry, F., Nissim, K. & Smith, A. (2006). Calibrating Noise to Sensitivity in Private Data Analysis. Theory of Cryptography Conference (TCC) 2006.
- Dwork, C. & Roth, A. (2014). The Algorithmic Foundations of Differential Privacy. Foundations and Trends in Theoretical Computer Science 9(3–4).
Mentioned in
AI securityAI security, n.: Protecting AI systems — their data, models and surrounding software — from being attacked, stolen or tricked, and dealing with the ways AI itself can be used to attack others., privacyprivacy, n.: Your ability to control who knows what about you. AI raises new privacy questions because it learns from huge amounts of personal data and can infer things about people they never shared., adversarial machine learningadversarial machine learning, n.: The study of how machine-learning systems can be fooled, corrupted or spied on by someone working deliberately against them, and of how to build systems that hold up., federated learningfederated learning, n.: Training one shared AI model across many phones, hospitals or companies without collecting their data in one place: each trains on its own data and sends back only model updates., General Data Protection RegulationGeneral Data Protection Regulation, prop. n.: The European Union's main privacy law, in force since 2018, which controls how personal information is collected and used — including to train and run AI systems., gradient clippinggradient clipping, n.: A safety limit on how big a single training update can be. If the computed adjustment is unusually large, it is scaled down so that one bad batch cannot wreck the model., machine unlearningmachine unlearning, n.: Erasing specific things a trained AI model has learned, such as dangerous lab techniques, a copyrighted novel or one person's records, far more cheaply than training the model all over again., membership inference attackmembership inference attack, n.: An attack that tries to tell whether a particular person's record, or a particular document, was part of an AI model's training data — which can itself reveal private facts., memorizationmemorization, n.: When an AI model stores specific pieces of its training data — a paragraph, a phone number, an image — closely enough to reproduce them, instead of only learning general patterns., personally identifiable informationpersonally identifiable information, n.: Information that can identify a specific person, such as a name, address, phone number or ID number — or several facts that together point to one person., privacy-preserving machine learningprivacy-preserving machine learning, n.: The family of techniques for training and using AI models while keeping the underlying personal or confidential data hidden — from adding careful noise to computing on encrypted data., training data extractiontraining data extraction, n.: Getting an AI model to repeat, word for word, pieces of the data it was trained on, which may include someone's private contact details or copyrighted text., canary stringcanary string, n.: A unique marker planted in data to trace what an AI model has absorbed: a code in test files so they can be kept out of training, or a fake secret hidden in training data to measure memorization., Cynthia DworkCynthia Dwork, prop. n.: An American computer scientist who co-invented differential privacy, the leading mathematical definition of privacy in data analysis, and helped found the study of fairness in algorithms., DP-SGDDP-SGD, abbr.: A way of training a neural network so that it cannot learn too much about any single person in the data: each example's influence is capped, and random noise is added at every step., gradient inversion attackgradient inversion attack, n.: Rebuilding someone's private training data — such as photos or text — just from the model updates their device shared during collaborative training., homomorphic encryptionhomomorphic encryption, n.: A way of locking data so that a computer can still work with it while it stays locked: the computer never sees the real numbers, and only the owner can open the answer., k-anonymityk-anonymity, n.: A rule for anonymizing tables of personal data: blur details such as age and postcode until every person matches at least k others, so no one can be singled out., local differential privacylocal differential privacy, n.: A privacy method in which each person's device scrambles their data with random noise before sending it anywhere, so the company collecting it never sees anyone's true answer but can still estimate overall trends., model inversionmodel inversion, n.: Using a trained AI model's answers to work backwards to private information it learned from — for example, rebuilding a recognizable picture of someone's face from a face-recognition system., PATEPATE, abbr.: A privacy method in which several 'teacher' models, each trained on a separate slice of sensitive data, vote noisily to label public examples, and a 'student' model learns only from those noisy votes., privacy budgetprivacy budget, n.: The total amount of privacy risk a data holder agrees to spend; each analysis or release uses up part of it, and once it is gone, no more answers can be given safely., randomized smoothingrandomized smoothing, n.: A way to make a classifier provably harder to fool: ask it about many noisy copies of an input and take a majority vote, which small deliberate changes cannot easily flip., secure aggregationsecure aggregation, n.: A cryptographic trick that lets a server add up updates from many devices while learning only the total, never any single device's contribution.