Nightshade
In plain wordsA free tool for artists that subtly alters their images so that AI image generators trained on them without permission learn the wrong things — associating 'dog' with pictures of cats, say.
A prompt-specific data poisoningdata poisoning, n.: Sneaking bad examples into the data an AI learns from, so the finished model behaves as the attacker wants — badly overall, or wrongly only in special situations. tool for text-to-image modelstext-to-image model, n.: An AI system that creates a picture from a written description, such as 'a fox reading a newspaper, in watercolor', usually within seconds., released by University of Chicago researchers for artists who object to their work being scraped for training. It perturbs an image so that people still see its original subject while a model's image encoder sees features of a different concept; because any single concept occupies a small share of training data, relatively few such images captioned with that concept can corrupt how a model renders it. It is meant as an offensive deterrent that raises the cost of training on unlicensed images, complementing the defensive GlazeGlaze, prop. n.: A free tool that lets artists treat their pictures before posting them, with changes people barely notice, so that an AI trained to imitate their style learns a different one.. Its effectiveness at web scale and against data filtering, image purification or more robust encoders is debated, as is the ethics of deliberately degrading shared models.
Several illustrators ran their portfolios through Nightshade before posting them online.
Origin
Developed by Shawn Shan, Wenxin Ding, Josephine Passananti, Stanley Wu, Haitao Zheng and Ben Zhao at the University of Chicago; described in an October 2023 preprint (IEEE Symposium on Security and Privacy 2024) and released publicly in January 2024. The name is that of a poisonous plant family.
Related
- GlazeGlaze, prop. n.: A free tool that lets artists treat their pictures before posting them, with changes people barely notice, so that an AI trained to imitate their style learns a different one.
- data poisoningdata poisoning, n.: Sneaking bad examples into the data an AI learns from, so the finished model behaves as the attacker wants — badly overall, or wrongly only in special situations.
- clean-label poisoningclean-label poisoning, n.: A sneaky form of data poisoning in which every planted training example is labeled correctly, so reviewers see nothing wrong, yet the examples are crafted to make the model fail.
- web scrapingweb scraping, n.: Using software to automatically copy information from websites — text, prices, images — in bulk, which is how much of the data used to train AI was gathered.
- AI copyright litigationAI copyright litigation, n.: The many lawsuits in which authors, artists, news organizations, record labels and others accuse AI companies of copying their work without permission to build or run AI models.
- text-to-image modeltext-to-image model, n.: An AI system that creates a picture from a written description, such as 'a fox reading a newspaper, in watercolor', usually within seconds.
Sources
- Shan, S., Ding, W., Passananti, J., Wu, S., Zheng, H. & Zhao, B. Y. (2024). Nightshade: Prompt-Specific Poisoning Attacks on Text-to-Image Generative Models. IEEE S&P 2024. arXiv:2310.13828.
Mentioned in
AI artAI art, n.: Images, music or other artworks made with the help of AI — from early computer drawing programs to today's prompt-based image generators — and the debate over whether and how such work counts as art., data poisoningdata poisoning, n.: Sneaking bad examples into the data an AI learns from, so the finished model behaves as the attacker wants — badly overall, or wrongly only in special situations., style transferstyle transfer, n.: Repainting a photo in the style of an artwork — your street as Van Gogh might have painted it — by keeping the photo's content and borrowing the painting's colors, strokes and textures., clean-label poisoningclean-label poisoning, n.: A sneaky form of data poisoning in which every planted training example is labeled correctly, so reviewers see nothing wrong, yet the examples are crafted to make the model fail., GlazeGlaze, prop. n.: A free tool that lets artists treat their pictures before posting them, with changes people barely notice, so that an AI trained to imitate their style learns a different one.