llms.txt
In plain wordsA short guide file that a website puts at a standard address to help AI assistants find and understand its most useful pages.
A proposed web convention: a site places a Markdown file called llms.txt at its root (or under a subpath) as a short, hand-picked guide for language modelslarge language model, n.: A computer program trained on enormous amounts of text to predict the words that come next. At vast scale, that simple skill lets it chat, answer questions, translate, summarize and write code. and agentsAI agent, n.: An AI system that does not just answer but acts: given a goal, it decides on steps, uses tools such as a browser or code editor, checks the results and keeps going until the job is done.. Only a top-level (H1) title is mandatory; it is normally followed by a one-paragraph blockquote summary, any further notes, and H2-headed lists of links, each with a brief description, while links under a heading named 'Optional' are ones an agent may drop when its contextcontext window, n.: How much an AI model can read and keep in view at once: the instructions, the conversation so far and any attached documents. Text that does not fit has to be cut, summarized or left out. is tight. The proposal also suggests serving a plain-Markdown copy of each useful page at its usual URL with '.md' added. It targets the moment an assistant is answering a user rather than model training, and so complements robots.txtrobots.txt, n.: A small public file in which a website posts its rules for automated visitors, such as search-engine and AI crawlers, listing areas they are asked to stay out of. Obeying it is voluntary., which controls crawler access, and sitemaps, which enumerate every page.
Point the agent at the library's llms.txt instead of letting it scrape the whole docs site.
Origin
Proposed by Jeremy Howard of Answer.AI on 3 September 2024 at llmstxt.org, and adopted by documentation platforms and many developer sites through 2025.
Usage note
A voluntary convention rather than a formal standard; how much individual AI crawlers and assistants actually use it varies.
Related
- robots.txtrobots.txt, n.: A small public file in which a website posts its rules for automated visitors, such as search-engine and AI crawlers, listing areas they are asked to stay out of. Obeying it is voluntary.
- generative engine optimizationgenerative engine optimization, n.: Shaping web content so that AI chatbots and AI search features are more likely to find it, quote it and recommend it — the AI-era version of search engine optimization.
- context engineeringcontext engineering, n.: Managing an AI model's reading material: choosing, trimming and arranging the instructions, documents, tool results and memories it is given on every turn, so that it has what the task needs and little else.
- web crawlerweb crawler, n.: A program that automatically travels the web, following links from page to page and downloading what it finds — used by search engines, archives and AI companies to gather data.
- agent skillagent skill, n.: A packaged how-to guide for an AI agent — written steps, sometimes with small programs and examples attached — that sits unread until a job of that kind comes up, when the agent opens it.
- progressive disclosureprogressive disclosure, n.: Showing only a short summary first and revealing details when they are actually needed. AI agents use it to avoid filling their limited memory with instructions they may never use.
Sources
- Howard, J. (2024). The /llms.txt file. llmstxt.org, 3 September 2024. llmstxt.org
Mentioned in
generative engine optimizationgenerative engine optimization, n.: Shaping web content so that AI chatbots and AI search features are more likely to find it, quote it and recommend it — the AI-era version of search engine optimization.