membership inference attack
In plain wordsAn attack that tries to tell whether a particular person's record, or a particular document, was part of an AI model's training data — which can itself reveal private facts.
A privacy attack that tries to establish, for a particular record or document, whether it was among the examples a model learned from, judging by how the model responds to that input and sometimes by its parameters. It exploits the gap between how models treat seen and unseen examples, typically lower loss or higher confidence on members. Early attacks trained 'shadow models' on similar data to learn this difference; modern ones calibrate per example, comparing the target model's loss with its distribution across reference models trained with and without the point (the likelihood-ratio attack), and are judged by true-positive rate at very low false-positive rates. Membership can itself be sensitive (a record in a cancer study), and membership inference is the standard empirical audit of differential privacydifferential privacy, n.: A mathematical promise that the results of analyzing a dataset — or a model trained on it — would look almost the same whether or not any one person's data was included, so little can be learned about that person. and memorizationmemorization, n.: When an AI model stores specific pieces of its training data — a paragraph, a phone number, an image — closely enough to reproduce them, instead of only learning general patterns.. It is also used to detect copyrighted or benchmark text in language modelslarge language model, n.: A computer program trained on enormous amounts of text to predict the words that come next. At vast scale, that simple skill lets it chat, answer questions, translate, summarize and write code..
At a 0.1% false-positive rate, the membership inference attack identified 8% of the training records.
Origin
Nils Homer and colleagues (2008) showed that an individual's presence in a genomic study could be inferred from published aggregate statistics. Reza Shokri, Marco Stronati, Congzheng Song and Vitaly Shmatikov introduced membership inference against machine-learning models, with shadow-model training, at the 2017 IEEE Symposium on Security and Privacy. Nicholas Carlini and colleagues' likelihood-ratio attack (2022) set the current evaluation standard.
Usage note
Contested for language-model pretraining data: 2024 studies found that many reported successes reflected distribution differences between member and non-member text rather than genuine inference.
Related
- differential privacydifferential privacy, n.: A mathematical promise that the results of analyzing a dataset — or a model trained on it — would look almost the same whether or not any one person's data was included, so little can be learned about that person.
- memorizationmemorization, n.: When an AI model stores specific pieces of its training data — a paragraph, a phone number, an image — closely enough to reproduce them, instead of only learning general patterns.
- training data extractiontraining data extraction, n.: Getting an AI model to repeat, word for word, pieces of the data it was trained on, which may include someone's private contact details or copyrighted text.
- model inversionmodel inversion, n.: Using a trained AI model's answers to work backwards to private information it learned from — for example, rebuilding a recognizable picture of someone's face from a face-recognition system.
- data contaminationdata contamination, n.: When the test questions used to grade an AI system were already in its training data, so a high score may reflect memorized answers rather than real ability.
Sources
- Shokri, R., Stronati, M., Song, C. & Shmatikov, V. (2017). Membership Inference Attacks Against Machine Learning Models. IEEE S&P 2017. arXiv:1610.05820.
- Carlini, N., Chien, S., Nasr, M., Song, S., Terzis, A. & Tramèr, F. (2022). Membership Inference Attacks From First Principles. IEEE S&P 2022.
Mentioned in
AI securityAI security, n.: Protecting AI systems — their data, models and surrounding software — from being attacked, stolen or tricked, and dealing with the ways AI itself can be used to attack others., privacyprivacy, n.: Your ability to control who knows what about you. AI raises new privacy questions because it learns from huge amounts of personal data and can infer things about people they never shared., adversarial machine learningadversarial machine learning, n.: The study of how machine-learning systems can be fooled, corrupted or spied on by someone working deliberately against them, and of how to build systems that hold up., data contaminationdata contamination, n.: When the test questions used to grade an AI system were already in its training data, so a high score may reflect memorized answers rather than real ability., differential privacydifferential privacy, n.: A mathematical promise that the results of analyzing a dataset — or a model trained on it — would look almost the same whether or not any one person's data was included, so little can be learned about that person., machine unlearningmachine unlearning, n.: Erasing specific things a trained AI model has learned, such as dangerous lab techniques, a copyrighted novel or one person's records, far more cheaply than training the model all over again., memorizationmemorization, n.: When an AI model stores specific pieces of its training data — a paragraph, a phone number, an image — closely enough to reproduce them, instead of only learning general patterns., privacy-preserving machine learningprivacy-preserving machine learning, n.: The family of techniques for training and using AI models while keeping the underlying personal or confidential data hidden — from adding careful noise to computing on encrypted data., training data extractiontraining data extraction, n.: Getting an AI model to repeat, word for word, pieces of the data it was trained on, which may include someone's private contact details or copyrighted text., black-box attackblack-box attack, n.: An attack on an AI model by someone who cannot see inside it and can only send it inputs and watch what comes out — like picking a lock by feel., Cynthia DworkCynthia Dwork, prop. n.: An American computer scientist who co-invented differential privacy, the leading mathematical definition of privacy in data analysis, and helped found the study of fairness in algorithms., DP-SGDDP-SGD, abbr.: A way of training a neural network so that it cannot learn too much about any single person in the data: each example's influence is capped, and random noise is added at every step., gradient inversion attackgradient inversion attack, n.: Rebuilding someone's private training data — such as photos or text — just from the model updates their device shared during collaborative training., k-anonymityk-anonymity, n.: A rule for anonymizing tables of personal data: blur details such as age and postcode until every person matches at least k others, so no one can be singled out., model inversionmodel inversion, n.: Using a trained AI model's answers to work backwards to private information it learned from — for example, rebuilding a recognizable picture of someone's face from a face-recognition system., privacy budgetprivacy budget, n.: The total amount of privacy risk a data holder agrees to spend; each analysis or release uses up part of it, and once it is gone, no more answers can be given safely.