A tiny text file with enormous power sits at the root of almost every website on the internet: robots.txt. Get it right, and search engines crawl your best pages efficiently while ignoring the clutter. Get it wrong, and you can accidentally block your entire site from Google overnight. This robots.txt guide explains exactly what the file is, how crawlers read it, and the mistakes that cost real websites their traffic.
Whether you run a blog, an agency site, or an ecommerce store, understanding robots.txt is a non-negotiable technical SEO skill. This robots.txt guide walks you through it step by step, from core syntax to the traps that cost real sites their traffic.
What Is Robots.txt?
A robots.txt file is a plain-text file placed in the root directory of a website. It gives crawling bots (also called spiders or crawlers), such as Googlebot or Bingbot, instructions about which parts of the site they may crawl and which parts they should skip. It is part of the Robots Exclusion Protocol, a voluntary standard that well-behaved crawlers respect.
Think of it as a doorman at a building entrance. Visitors can still see the building from the street, but the doorman tells them which floors are open and which are off-limits. Importantly, robots.txt controls crawling (whether a bot fetches a page), not indexing (whether a page appears in search results). That distinction is the source of most confusion, and we will return to it later in this robots.txt guide.
The file itself is remarkably simple. A basic example looks like this:
- User-agent: *
- Disallow: /admin/
- Sitemap: https://yourdomain.com/sitemap.xml
Those three lines tell every crawler to stay out of the /admin/ folder and point them at your XML sitemap. Simple, but every line matters, and the rest of this robots.txt guide shows you how to get each one right.
How Do Search Crawlers Read Robots.txt?
Before a search engine bot crawls anything on your site, it checks for your robots.txt file at the root, for example yourdomain.com/robots.txt. It downloads the file, finds the rule group that matches its own user-agent name, and follows the most specific matching rule for every URL it wants to fetch.
Here is the logic crawlers apply:
- Locate the matching group: A crawler identifies itself with a user-agent string. It looks for a group named for it (for example Googlebot). If none exists, it uses the group for *, which means all crawlers.
- Apply the most specific rule: When an Allow and a Disallow rule both match a URL, the longer, more specific path wins, regardless of order.
- Default to allow: Anything not matched by a rule is crawlable. Robots.txt is an opt-out system, not an opt-in one.
- Cache the file: Crawlers cache robots.txt and re-fetch it periodically, typically about every 24 hours, so changes are not instant.
One crucial detail this robots.txt guide cannot skip: if a crawler cannot reach your robots.txt file because of a server error, Googlebot will stop crawling the entire site rather than risk violating rules it cannot see. A 500 error on this little file can freeze your crawl completely, which is why monitoring matters.
A Robots.txt Guide to the Core Directives
Every robots.txt guide must cover the four building blocks of the protocol. Here they are, with what each one does:
| Directive | What it does | Example |
|---|---|---|
| User-agent | Names the crawler a group of rules applies to. An asterisk targets all crawlers. | User-agent: Googlebot |
| Disallow | Lists URL paths the crawler should not fetch. A trailing slash blocks everything beneath. | Disallow: /private/ |
| Allow | Carves out an exception inside a disallowed area so specific files stay crawlable. | Allow: /private/report.pdf |
| Sitemap | Declares the location of your XML sitemap so crawlers can discover your pages faster. | Sitemap: https://yourdomain.com/sitemap.xml |
Google and Bing also support two wildcards. The asterisk * matches any sequence of characters, and the dollar sign $ marks the end of a URL. For example, Disallow: /*.pdf$ blocks every PDF file on your site, while Disallow: /search? blocks dynamic search-result pages that generate endless crawl waste.
A clean, safe starting template for most small sites is:
- User-agent: *
- Allow: /
- Disallow: /wp-admin/
- Disallow: /*?sort=
- Sitemap: https://yourdomain.com/sitemap.xml
Notice what it does not block: CSS, JavaScript, and image folders. Google needs those resources to render your pages properly, and blocking them can distort how your site appears in search results.
Where Does Robots.txt Live and How Do You Create One?
The file must live at the root of your domain, reachable at yourdomain.com/robots.txt. A file placed in a subfolder, such as yourdomain.com/blog/robots.txt, is invisible to crawlers. Each subdomain and each protocol (HTTP vs HTTPS) needs its own file, a detail every robots.txt guide must mention.
- Open a plain-text editor such as Notepad or VS Code. Never use Word or Google Docs, which insert formatting characters that corrupt the file.
- Write your rules, starting with a User-agent line, then your Disallow and Allow lines, then the Sitemap line.
- Save the file as exactly robots.txt, encoded in UTF-8.
- Upload it to the root directory of your web server.
- Visit yourdomain.com/robots.txt in a browser to confirm it loads.
- Test it with the robots.txt tester in Google Search Console before you relax.
If you use WordPress, your SEO plugin (Rank Math, Yoast, or similar) can generate and edit the file for you, but always verify the public URL afterwards. A physical file in the server root overrides whatever a plugin generates, and CDN rules can override both. If this robots.txt guide has one mantra, it is this: verify the live file, not the plugin preview.
10 Common Robots.txt Mistakes to Avoid
This section is the heart of this robots.txt guide. These are the errors that real sites make, ranked by how much damage they cause.
1. Blocking Your Entire Site by Accident
The single most destructive line in SEO is User-agent: * followed by Disallow: /. It tells every crawler to leave your whole site alone. This typically leaks from a staging environment into production, or appears when someone means to block one folder and forgets the folder name. Always double-check before deploying.
2. Blocking CSS, JavaScript, and Images
Google renders pages to understand layout, ads, and content positioning. If you disallow your theme, plugin, or media folders, Google cannot render your pages, which can hurt rankings. Never block paths that serve CSS, JS, fonts, or images.
3. Using Robots.txt as a Security Tool
Robots.txt is public. Anyone can read it, and a Disallow line is effectively a signpost pointing at whatever you named. Malicious bots ignore it entirely. Protect sensitive areas with authentication, server rules, or IP restrictions, never with robots.txt alone.
4. Trying to Deindex Pages With Disallow
Disallow prevents crawling, but a blocked page can still appear in Google if other sites link to it, usually without a snippet. If you want a page out of search results, use a noindex meta tag or X-Robots-Tag header, and leave the page crawlable so Google can see the directive. Blocking it hides the noindex tag from Google, which is the worst of both worlds.
5. Forgetting the Sitemap Line
The Sitemap directive is optional, but skipping it slows discovery. It gives crawlers a direct map to your most important pages instead of forcing them to find everything through links alone. One line, free efficiency.
6. Ignoring Case Sensitivity
Paths in robots.txt are case-sensitive. Blocking /Admin/ does nothing to protect /admin/. Always match the exact casing of your real URLs, or you will block the wrong thing and miss the right one.
7. Bad Syntax and Typos
A missing colon after User-agent, a stray space before a slash, or a duplicated directive can break a rule or the entire file. Keep one space after each colon, one directive per line, and test every edit. Robots.txt is not forgiving.
8. Assuming Groups Merge
If you create a group for Googlebot, Googlebot ignores the generic * group entirely. Every rule Googlebot still needs must be repeated inside its own group. This surprises even experienced SEOs and silently drops protections or sitemap declarations.
9. Using Crawl-delay for Google
Googlebot ignores Crawl-delay completely. Bing and Yandex respect it. If Google is crawling too aggressively, use the crawl-rate controls in Google Search Console instead of a directive Google will never read.
10. Never Testing or Re-auditing the File
Sites change: new sections launch, developers push staging configs, plugins rewrite the file. A rule that was safe six months ago can become a disaster after a redesign. Schedule a quick robots.txt review after every major site change, and keep an eye on the crawl stats in Search Console.
How to Test Your Robots.txt File
Writing the file is only half the job. This robots.txt guide recommends verifying it with these steps before you consider it done:
- Google Search Console robots.txt tester: submit a URL and see whether Googlebot is allowed or blocked, then test edits without publishing them.
- Fetch in a private window: open yourdomain.com/robots.txt to confirm the live content matches what you intended.
- Crawl simulation: tools like Screaming Frog can crawl your site while respecting your rules, revealing pages that are unexpectedly blocked.
- Coverage report: check Search Console for a spike in “Blocked by robots.txt” exclusions, the classic symptom of an accidental block.
As search evolves, crawler management keeps getting more interesting. AI answer engines and crawlers such as GPTBot or ClaudeBot are now part of the conversation, and the same file controls whether they may train on or cite your content. For a deeper look at how search is shifting, read our guide on how to rank in Google AI Overviews, and if you are curious about the newer machine-readable file for AI agents, see what llms.txt is and why it matters.
For the authoritative specification, Google documents the full protocol in its robots.txt introduction for Search Central, and the protocol itself is standardized in RFC 9309.
FAQs
These are the questions this robots.txt guide hears most often, drawn from real search queries and community discussions.
Can robots.txt keep a page out of Google search results?
Not reliably. As this robots.txt guide explains, robots.txt asks crawlers not to fetch a page, but Google can still index the URL if other websites link to it, usually displaying it without a description. To genuinely remove a page from results, use a noindex meta tag or X-Robots-Tag header and keep the page crawlable so Google can process that directive.
What happens if my website has no robots.txt file at all?
Nothing catastrophic. Crawlers simply treat the whole site as crawlable, which is the default behavior anyway. A missing file is not an error. That said, adding one with a Sitemap declaration is still worthwhile, because it speeds up discovery of your important pages and gives you a control lever when you need it, which is why this robots.txt guide recommends a minimal file even for small sites.
Should I block AI crawlers like GPTBot in robots.txt?
It depends on your goals. Blocking training crawlers such as GPTBot or ClaudeBot keeps your content out of AI model training data, which some publishers prefer. Leaving AI search crawlers allowed keeps you eligible to be cited and linked in AI-generated answers. Decide by purpose, and name each crawler explicitly rather than relying on broad blocks, as this robots.txt guide advises.
How quickly do robots.txt changes take effect?
Crawlers cache robots.txt and re-fetch it roughly every 24 hours, so allow at least a day for changes to propagate across all bots. If you have just removed an accidental site-wide block, this robots.txt guide suggests using Google Search Console to request revalidation and recrawling of your most important pages rather than waiting passively.
Is it mandatory to have a robots.txt file?
No, it is not mandatory, and most small sites function perfectly well without one. It becomes valuable when you need to manage crawl budget, keep crawlers out of admin or staging areas, block parameter-generated URLs, or declare your sitemap. If none of those apply, a minimal file with a Sitemap line is still good practice — the same advice this robots.txt guide gives to every new site.
Does blocking a page in robots.txt also block its CSS and JS?
Only if your rules are written that broadly. Disallowing a folder blocks every file beneath it, including stylesheets and scripts, unless you add Allow exceptions. This is why you should never block entire asset directories, which is exactly why this robots.txt guide stresses testing every change in Search Console.
Robots.txt is one of the smallest files on your site and one of the most consequential, which is exactly why this robots.txt guide exists. A few careful lines protect your crawl budget, keep the wrong pages out of bots’ way, and let your best content shine. Review your file today, and if you want an expert eye on your whole technical setup, get a free site audit at ssaagency.us.