robots.txt
In plain wordsA small public file in which a website posts its rules for automated visitors, such as search-engine and AI crawlers, listing areas they are asked to stay out of. Obeying it is voluntary.
A plain-text file at the root of a website that states, per crawler user agent, which URL paths automated clients may or may not access, under the Robots Exclusion Protocol. Rules are grouped under 'User-agent' lines with 'Allow' and 'Disallow' path patterns; crawlers are expected to fetch and cache the file and apply the most specific matching rule. The protocol is a convention, not access control: it depends on crawler cooperation, and the paths it lists become public. Since 2023 it has been the main way publishers refuse AI training: AI developers name distinct user agents, or control tokens that do no crawling themselves, so that sites can disallow training while still allowing search indexing. Refusals spread quickly: a 2024 audit of the sources behind the C4C4, prop. n.: A large collection of English web text, cleaned up from a 2019 web archive snapshot by Google researchers, that became a standard ingredient for training language models. corpus found that within one year (2023–24) full restrictions had come to cover at least 5% of all its tokens and at least 28% of its most frequently updated, high-value sources.
Adding a Disallow rule for the training crawler kept the site in search results but out of future crawls.
Origin
Proposed by Martijn Koster, then at the software firm Nexor, on the www-talk mailing list in February 1994; the writer Charles Stross has said that a badly behaved crawler of his, which overloaded Koster's server, prompted the idea. It remained an informal convention for 28 years until the IETF published it as RFC 9309 in September 2022 (Koster, Gary Illyes, Henner Zeller and Lizzi Sassman).
Usage note
robots.txt says nothing about how fetched content may be used, cannot bind crawlers that ignore it, and does not reach data collected earlier. Complementary schemes include machine-readable licensing terms such as Really Simple Licensing (2025) and the EU's text-and-data-mining reservation; see opt-outopt-out, n.: A way for creators, websites or individuals to say 'do not use my work or data to train AI' — instead of being asked for permission first, which would be opt-in..
Related
- web crawlerweb crawler, n.: A program that automatically travels the web, following links from page to page and downloading what it finds — used by search engines, archives and AI companies to gather data.
- web scrapingweb scraping, n.: Using software to automatically copy information from websites — text, prices, images — in bulk, which is how much of the data used to train AI was gathered.
- opt-outopt-out, n.: A way for creators, websites or individuals to say 'do not use my work or data to train AI' — instead of being asked for permission first, which would be opt-in.
- Common CrawlCommon Crawl, prop. n.: A nonprofit's free archive of the public web, begun in 2008 and now refreshed with snapshots of billions of pages every month or two. Most large language models learned much of what they know from text taken from it.
- data licensingdata licensing, n.: The legal permissions that say who may use a dataset or a body of content, for what purposes and on what terms — including whether it may be used to train AI.
- text and data mining exceptiontext and data mining exception, n.: A rule in many countries' copyright laws that lets people use computers to copy and analyze large amounts of text, images or data without asking permission — sometimes only for research, sometimes unless owners opt out.
Sources
- Koster, M., Illyes, G., Zeller, H. & Sassman, L. (2022). Robots Exclusion Protocol. IETF RFC 9309. rfc-editor.org/rfc/rfc9309
- Longpre, S. et al. (2024). Consent in Crisis: The Rapid Decline of the AI Data Commons. arXiv:2407.14933. arxiv.org/abs/2407.14933
Defined in law or standards
IETF RFC 9309, Robots Exclusion Protocol (Proposed Standard, September 2022)
Mentioned in
robotrobot, n.: A machine that senses its surroundings, decides what to do and physically acts, moving itself or handling objects, with some independence from a human operator. Factory arms, robot vacuums and self-driving cars are all robots., Common CrawlCommon Crawl, prop. n.: A nonprofit's free archive of the public web, begun in 2008 and now refreshed with snapshots of billions of pages every month or two. Most large language models learned much of what they know from text taken from it., data licensingdata licensing, n.: The legal permissions that say who may use a dataset or a body of content, for what purposes and on what terms — including whether it may be used to train AI., opt-outopt-out, n.: A way for creators, websites or individuals to say 'do not use my work or data to train AI' — instead of being asked for permission first, which would be opt-in., text and data mining exceptiontext and data mining exception, n.: A rule in many countries' copyright laws that lets people use computers to copy and analyze large amounts of text, images or data without asking permission — sometimes only for research, sometimes unless owners opt out., web crawlerweb crawler, n.: A program that automatically travels the web, following links from page to page and downloading what it finds — used by search engines, archives and AI companies to gather data., web scrapingweb scraping, n.: Using software to automatically copy information from websites — text, prices, images — in bulk, which is how much of the data used to train AI was gathered., llms.txtllms.txt, n.: A short guide file that a website puts at a standard address to help AI assistants find and understand its most useful pages.