Robots.txt Guide: How It Works and Common Mistakes
A robots.txt guide based on Google’s specification: syntax, rule precedence, server-error behavior, testing, AI crawler controls, and the mistakes to avoid.
What Is llms.txt? The Beginner’s Guide to the New AI Crawler File
You have probably heard of robots.txt — the small file that tells search engine crawlers which parts of your website they may visit. Now there is a new file catching the attention of website owners in the age of AI search: llms.txt. So what is llms.txt, and does your small business need one? In short, llms.txt is a simple text file that gives AI models a clean, curated tour of your most important pages — instead of letting an AI assistant guess what your business offers from raw HTML. This guide explains what the file does, how it differs from robots.txt, what the honest evidence says about its effectiveness in 2026, and how to create one in about 20 minutes. What Is llms.txt? A Simple Explanation llms.txt is a plain Markdown file published at your domain’s root — for example, yourdomain.com/llms.txt — that lists your site’s most important pages with short, human-written descriptions. Think of it as a cheat sheet for AI models: your pricing page, your service pages, your documentation, and your key FAQs, all organized in one tidy document. The name comes from LLM, which stands for large language model — the technology behind AI assistants like ChatGPT, Claude, and Perplexity. When someone asks one of these assistants about businesses like yours, it needs accurate, current information. Today most models piece that together from raw HTML — messy code full of menus, popups, and layout clutter — while the file offers a cleaner alternative: the pages you want them to see, described in your own words. One important point up front: the file is a proposed convention, not a ratified formal standard. A group of practitioners has suggested it as a best practice and a growing number of sites have adopted it — but no official standards body has approved it, and no AI company must read it. Where Did llms.txt Come From? The idea was proposed by Jeremy Howard — co-founder of Answer.AI and creator of the machine-learning course fast.ai — on September 3, 2024 (read the original proposal). His argument was simple: large language models are becoming a major way people find information online, yet websites offer them nothing better than raw HTML. He suggested a short Markdown file as the fix. The proposal was revised as version 2 on August 10, 2026, keeping the format short and adding flexibility — allowing the file in a subpath like /docs/llms.txt. Adoption has grown mostly among documentation-heavy sites and developer tools. How llms.txt Works: What’s Inside the File The format is short by design and written in Markdown — the simple formatting style used in many blogging tools. Here is what it contains: The one required section The file must start with an H1 heading that names your site or project — the only strictly required part. For example: # Bright Smile Dental Studio. This single line tells any AI model reading the file exactly whose content it is looking at. The recommended summary Right after the heading comes a blockquote — a single quoted line summarizing your site in one sentence. In Markdown, a blockquote starts with a > symbol. For example: > Family dental practice in Austin offering same-day appointments and transparent pricing. Optional context paragraphs Next, you can add a few short paragraphs of background: who you serve, what makes you different, and what a visitor can do on your site. Keep it to two or three sentences — the whole point of llms.txt is brevity. H2 sections with link lists The heart of the file is a set of H2 sections, each containing a list of Markdown links with short descriptions. Typical sections include Services, Company, Pricing, and Resources. A link entry shows a page title in square brackets, the URL in parentheses, and a one-line description after a colon. There is also an optional companion file, /llms-full.txt, which can hold full copies of your important content. Most small businesses can skip it and publish just the shorter index. llms.txt vs robots.txt vs sitemap.xml: What’s the Difference? New website files can blur together, so let us draw sharp lines: robots.txt is about access control, while llms.txt is about content description. They solve opposite problems and are meant to be used together. robots.txt tells crawlers where they may and may not go — think of it as a “keep out” sign with allow and deny rules. llms.txt does the opposite: it is an index and an invitation that says “here is what our site is about,” controlling and enforcing nothing. Listing a page in llms.txt does not open that page to blocked crawlers, and leaving one out does not protect it — access is still governed entirely by robots.txt. sitemap.xml is a third, separate file: a machine-readable list of URLs that helps crawlers discover and index your pages. It says “these pages exist” but nothing about what they mean. See the comparison below: Feature robots.txt sitemap.xml llms.txt Purpose Controls which crawlers may visit which pages Lists URLs so crawlers can discover pages Describes your key pages for AI models Who reads it Search engine and AI crawlers Search engine crawlers AI models and AI assistants Format Plain text rules (allow / disallow) XML list of URLs Markdown with headings and links Controls access? Yes — grants or denies crawling No No — describes only Helps discovery? Indirectly, by guiding crawlers Yes — its main job Yes — points AI to key pages Helps understanding? No No Yes — human-written descriptions Official standard? Yes — widely adopted convention Yes — supported by all major search engines No — proposed convention only Does llms.txt Actually Work in 2026? An Honest Look Any agency that tells you this file will boost your traffic tomorrow is selling you something. Here is the honest picture as of 2026: Google says Google Search ignores these files. Google has confirmed they neither help nor harm your visibility in search results. Adding one will not move your rankings, and skipping one will