web scraping
In plain wordsUsing software to automatically copy information from websites — text, prices, images — in bulk, which is how much of the data used to train AI was gathered.
The automated extraction of content from web pages by programs that fetch them (see web crawlerweb crawler, n.: A program that automatically travels the web, following links from page to page and downloading what it finds — used by search engines, archives and AI companies to gather data.) and parse their HTML, rendered output or underlying APIs to pull out text, images, prices, profiles or other structured data. Scraping supplies much of the data behind AI: web-scale text corpora derived from Common CrawlCommon Crawl, prop. n.: A nonprofit's free archive of the public web, begun in 2008 and now refreshed with snapshots of billions of pages every month or two. Most large language models learned much of what they know from text taken from it., image–caption datasets such as LAION-5BLAION-5B, prop. n.: A free list of nearly six billion links to online images paired with their captions, released in 2022 and used to train image generators such as Stable Diffusion., code from public repositories, and the live pages fetched by AI search and browsing agents. Its legality depends on jurisdiction and circumstance: copyright, database rights, website terms of service, computer-misuse statutes, data-protection law for personally identifiable informationpersonally identifiable information, n.: Information that can identify a specific person, such as a name, address, phone number or ID number — or several facts that together point to one person., and technical signals such as robots.txtrobots.txt, n.: A small public file in which a website posts its rules for automated visitors, such as search-engine and AI crawlers, listing areas they are asked to stay out of. Obeying it is voluntary. all bear on it. The surge of AI crawling since 2023 has led publishers to block AI bots, sue, or sell licensed access instead (see data licensingdata licensing, n.: The legal permissions that say who may use a dataset or a body of content, for what purposes and on what terms — including whether it may be used to train AI.).
We scraped product reviews from public pages, honoring robots.txt and rate limits.
Origin
'Screen scraping' first meant capturing text from the terminal displays of legacy systems; web scraping arrived with the first web robots in 1993. In the US, eBay v. Bidder's Edge (2000) treated aggressive scraping as trespass to chattels; in hiQ Labs v. LinkedIn the Ninth Circuit held at the preliminary-injunction stage (2019, reaffirmed 2022) that scraping public pages likely did not violate the Computer Fraud and Abuse Act, but the trial court later found that hiQ had breached LinkedIn's user agreement, and the parties settled in December 2022.
Usage note
Scraping and crawling are often conflated: crawling discovers and fetches pages; scraping extracts data from them. Whether scraping copyrighted material for AI training is lawful is being litigated in many countries (see AI copyright litigationAI copyright litigation, n.: The many lawsuits in which authors, artists, news organizations, record labels and others accuse AI companies of copying their work without permission to build or run AI models. and text and data mining exceptiontext and data mining exception, n.: A rule in many countries' copyright laws that lets people use computers to copy and analyze large amounts of text, images or data without asking permission — sometimes only for research, sometimes unless owners opt out.).
Related
- web crawlerweb crawler, n.: A program that automatically travels the web, following links from page to page and downloading what it finds — used by search engines, archives and AI companies to gather data.
- robots.txtrobots.txt, n.: A small public file in which a website posts its rules for automated visitors, such as search-engine and AI crawlers, listing areas they are asked to stay out of. Obeying it is voluntary.
- Common CrawlCommon Crawl, prop. n.: A nonprofit's free archive of the public web, begun in 2008 and now refreshed with snapshots of billions of pages every month or two. Most large language models learned much of what they know from text taken from it.
- data licensingdata licensing, n.: The legal permissions that say who may use a dataset or a body of content, for what purposes and on what terms — including whether it may be used to train AI.
- opt-outopt-out, n.: A way for creators, websites or individuals to say 'do not use my work or data to train AI' — instead of being asked for permission first, which would be opt-in.
- fair use and AI trainingfair use and AI training, phr.: The unsettled US legal question of whether AI companies may copy books, articles, pictures and music to train their models without permission, under the rule that allows some unlicensed uses of copyrighted work.
Mentioned in
CAPTCHACAPTCHA, n.: A small test on websites, like typing wavy letters or clicking every picture with a traffic light, meant to be easy for people but hard for computer programs, so as to keep automated bots out., Common CrawlCommon Crawl, prop. n.: A nonprofit's free archive of the public web, begun in 2008 and now refreshed with snapshots of billions of pages every month or two. Most large language models learned much of what they know from text taken from it., robots.txtrobots.txt, n.: A small public file in which a website posts its rules for automated visitors, such as search-engine and AI crawlers, listing areas they are asked to stay out of. Obeying it is voluntary., text and data mining exceptiontext and data mining exception, n.: A rule in many countries' copyright laws that lets people use computers to copy and analyze large amounts of text, images or data without asking permission — sometimes only for research, sometimes unless owners opt out., web crawlerweb crawler, n.: A program that automatically travels the web, following links from page to page and downloading what it finds — used by search engines, archives and AI companies to gather data., Clearview AIClearview AI, prop. n.: A US company that collected billions of photos from the internet without permission to build a face-search tool for police, the best-known example of controversial facial recognition., LAION-5BLAION-5B, prop. n.: A free list of nearly six billion links to online images paired with their captions, released in 2022 and used to train image generators such as Stable Diffusion., NightshadeNightshade, prop. n.: A free tool for artists that subtly alters their images so that AI image generators trained on them without permission learn the wrong things — associating 'dog' with pictures of cats, say.