web crawler
In plain wordsA program that automatically travels the web, following links from page to page and downloading what it finds — used by search engines, archives and AI companies to gather data.
A program that explores the web unattended: it fetches a page, extracts its links and adds any new addresses to a queue of pages still to visit (the frontier), so that a few starting URLs (seeds) can lead it across much of the web. Production crawlers schedule revisits, avoid duplicates and traps, spread their load across sites, identify themselves with a user-agent string and are expected to obey robots.txtrobots.txt, n.: A small public file in which a website posts its rules for automated visitors, such as search-engine and AI crawlers, listing areas they are asked to stay out of. Obeying it is voluntary. and rate limits. Search engines and archives have crawled since the 1990s; Common CrawlCommon Crawl, prop. n.: A nonprofit's free archive of the public web, begun in 2008 and now refreshed with snapshots of billions of pages every month or two. Most large language models learned much of what they know from text taken from it.'s CCBot made periodic whole-web snapshots freely available; and since 2023 AI developers have run their own crawlers, such as OpenAI's GPTBot and Anthropic's ClaudeBot, to gather pretraining datapretraining data, n.: The enormous body of text, code and other material — mostly from the web — that a large AI model first learns from, before it is tuned to follow instructions or chat., alongside on-demand fetchers that retrieve pages for AI search and agents. The resulting load on websites and disputes over consent have led many sites to block AI user agents, and some network providers to block them by default or charge for access.
The crawler respects robots.txt and backs off when a site returns rate-limit errors.
Origin
Matthew Gray's World Wide Web Wanderer (MIT, June 1993) is usually counted as the first web robot, built to measure the web's growth; JumpStation (December 1993) was the first crawler-based search engine, and Brian Pinkerton's WebCrawler (University of Washington, April 1994) was the first to index the full text of pages. 'Spider' extends the web metaphor.
Usage note
Distinguish crawling (finding and fetching pages) from web scrapingweb scraping, n.: Using software to automatically copy information from websites — text, prices, images — in bulk, which is how much of the data used to train AI was gathered. (extracting data from them), and bulk training crawlers from user-triggered fetchers, which some AI companies say are not bound by robots.txt rules written for crawlers.
Related
- web scrapingweb scraping, n.: Using software to automatically copy information from websites — text, prices, images — in bulk, which is how much of the data used to train AI was gathered.
- robots.txtrobots.txt, n.: A small public file in which a website posts its rules for automated visitors, such as search-engine and AI crawlers, listing areas they are asked to stay out of. Obeying it is voluntary.
- Common CrawlCommon Crawl, prop. n.: A nonprofit's free archive of the public web, begun in 2008 and now refreshed with snapshots of billions of pages every month or two. Most large language models learned much of what they know from text taken from it.
- information retrievalinformation retrieval, n.: The study and engineering of search: how to organize huge piles of documents so that the few that matter to someone's question can be found and ranked quickly. AI systems rely on it to look things up.
- pretraining datapretraining data, n.: The enormous body of text, code and other material — mostly from the web — that a large AI model first learns from, before it is tuned to follow instructions or chat.
- opt-outopt-out, n.: A way for creators, websites or individuals to say 'do not use my work or data to train AI' — instead of being asked for permission first, which would be opt-in.
Mentioned in
Common CrawlCommon Crawl, prop. n.: A nonprofit's free archive of the public web, begun in 2008 and now refreshed with snapshots of billions of pages every month or two. Most large language models learned much of what they know from text taken from it., data pipelinedata pipeline, n.: The chain of automated steps that collects raw data, cleans and transforms it, and delivers it in a usable form — the factory line that turns messy inputs into training material., opt-outopt-out, n.: A way for creators, websites or individuals to say 'do not use my work or data to train AI' — instead of being asked for permission first, which would be opt-in., robots.txtrobots.txt, n.: A small public file in which a website posts its rules for automated visitors, such as search-engine and AI crawlers, listing areas they are asked to stay out of. Obeying it is voluntary., web scrapingweb scraping, n.: Using software to automatically copy information from websites — text, prices, images — in bulk, which is how much of the data used to train AI was gathered., LLM groomingLLM grooming, n.: Flooding the internet with propaganda written mainly for machines rather than people, hoping AI chatbots will soak it up and repeat it to their users., llms.txtllms.txt, n.: A short guide file that a website puts at a standard address to help AI assistants find and understand its most useful pages.