11 min read · Technical SEO · Last updated July 2026
Quick answer: robots.txt tells crawlers which pages to skip. It prevents crawling but does not prevent indexing — a page can still appear in Google’s index if other sites link to it, even if you disallow it. Use it to block wasteful URLs, not sensitive content.
Introduction
A single line in robots.txt can wipe out thousands of pages from Google’s index. We’ve seen it happen — a developer adds Disallow: / to block staging, forgets to remove it before launch, and the live site drops out of search results within days.
robots.txt is a deceptively simple file. The syntax looks straightforward, but the edge cases, wildcard rules, and the crucial distinction between “blocking crawling” and “blocking indexing” trip up experienced developers and SEOs constantly.
This guide covers everything you need to know to write a correct, effective robots.txt file and avoid the mistakes that accidentally kill your organic traffic.
What you’ll learn:
– The exact syntax for user-agent, disallow, allow, and crawl-delay directives
– How wildcard rules work and when they apply
– The critical difference between blocking crawling vs. blocking indexing
– Common robots.txt mistakes and how to test your configuration
Table of Contents
- What robots.txt Does (and Doesn’t Do)
- robots.txt Syntax: User-Agent, Disallow, Allow
- Wildcard Rules: Matching URL Patterns
- Crawl-Delay: Use With Caution
- What to Block and What to Leave Open
- robots.txt vs. Noindex: Choosing the Right Tool
- Testing Your robots.txt Configuration
- The 10 Most Dangerous robots.txt Mistakes
What robots.txt Does (and Doesn’t Do)
robots.txt lives at yourdomain.com/robots.txt. It’s the first thing most crawlers check before exploring your site. The file follows the Robots Exclusion Protocol — a standard, not an enforced rule.
What it does:
– Instructs crawlers which URLs they should not crawl
– Reduces server load from bot traffic
– Controls crawl budget by steering bots away from low-value URLs
What it does NOT do:
– Prevent a page from being indexed (Google can index pages it’s never crawled if other sites link to them)
– Secure sensitive content (any user can read your robots.txt and see what you’re hiding)
– Apply to all bots (only bots that choose to follow the protocol respect it — malicious scrapers ignore it)
This is the most misunderstood aspect of robots.txt. Blocking Disallow: /admin/ does not hide your admin panel from the internet — it just tells well-behaved crawlers not to visit it. Use authentication to protect sensitive URLs, not robots.txt.
robots.txt Syntax: User-Agent, Disallow, Allow
Every robots.txt file is made of records. Each record has a User-agent specifying which bot the rules apply to, followed by Disallow and/or Allow directives.
User-agent: *
Disallow: /search/
Disallow: /account/
Allow: /
User-agent: Googlebot
Disallow: /no-google/
Sitemap: https://example.com/sitemap.xml
Key syntax rules:
User-agent: *applies to all crawlersUser-agent: Googlebotapplies only to Google’s crawlerDisallow: /path/blocks all URLs starting with/path/Disallow: /blocks the entire site (be careful)Allow: /permits crawling (useful to override a disallow)- Empty
Disallow:means allow everything - Comments start with
#
Rules within a record are evaluated in order, with the most specific matching rule winning. If you have Disallow: / and Allow: /public/, Google will crawl /public/ because it’s more specific.
Wildcard Rules: Matching URL Patterns
Google (and Bing) support two wildcards in robots.txt:
* — matches any sequence of characters
Disallow: /*?
Blocks any URL containing a ? (query string) anywhere in the path.
Disallow: /*.pdf$
Wait — this has a $ at the end. That’s the second wildcard.
$ — matches the end of the URL
Disallow: /*.pdf$
Blocks URLs ending in .pdf. Without the $, /pdf/info would also be matched.
Combining wildcards:
Disallow: /products/*?color=
Disallow: /search/?q=
Disallow: /*sessionid=
Note: * in the user-agent line means “all bots.” Wildcards in the Disallow/Allow lines mean pattern matching in URLs. These are different uses of the same character.
Not all crawlers support wildcards. Google, Bing, and most major crawlers do. The original robots exclusion standard only defined exact path matching — wildcards are an extension adopted by major search engines but not universally required.
Wildcard Pattern Tester
robots.txt Rule Tester
Test a Disallow rule against any URL path to see if it matches.
Crawl-Delay: Use With Caution
The Crawl-delay directive tells crawlers how many seconds to wait between requests:
User-agent: *
Crawl-delay: 10
Important: Google does not support Crawl-delay in robots.txt. Googlebot uses its own crawl rate determination based on server response times and your GSC crawl rate settings.
Crawl-delay is respected by Bing, Yandex, and other crawlers. If you need to slow down Googlebot specifically, use the crawl rate settings in Google Search Console (Settings → Crawl Rate).
Setting an excessively high crawl-delay (30+ seconds) to manage server load is usually a sign that your server is underpowered or poorly cached. Fix the underlying server problem rather than throttling crawlers.
What to Block and What to Leave Open
Block these:
– Admin and CMS backend URLs: /wp-admin/, /admin/, /dashboard/
– Internal search result pages: /search?q=
– Parameter-generated duplicates: /*?sort=, /*?filter=, /*?page=*&size=
– Staging and development paths: /staging/, /dev/, /test/
– Thank-you and confirmation pages: /thank-you/, /order-confirmation/
– Login and registration pages (if no indexation value): /login, /register
– API endpoints: /api/
– Print-friendly versions: /*?print=, /print/
Leave open (or explicitly allow):
– All product, service, category, and landing pages
– Blog content and articles
– Sitemap file (and ensure it’s listed in robots.txt)
– CSS and JavaScript files (critical for rendering)
– Image and media files linked from indexable pages
robots.txt vs. Noindex: Choosing the Right Tool
This distinction is critical and gets confused constantly.
| Situation | Use |
|---|---|
| Page should never be crawled | robots.txt Disallow |
| Page should be crawled but not indexed | Meta noindex tag |
| Page has thin content but some internal value | Meta noindex |
| Parameter URL (low value) | robots.txt Disallow |
| Staging environment | robots.txt Disallow |
| Paginated series beyond page 2 | Meta noindex |
| Admin/backend pages | robots.txt Disallow |
Never use robots.txt to block a page and also add a noindex to that page. If the page is disallowed, Google can’t read the noindex tag. Google may still index the URL from external links — without noindex to tell it not to.
If a page must not appear in search results under any circumstances, both disallow AND noindex are less effective than disallow + noindex. The right approach: if you can’t crawl it, Google can’t read the noindex. So either allow crawling with noindex, or block crawling and accept that linked pages might still show up as URL-only results (no snippet).
The 10 Most Dangerous robots.txt Mistakes
Common robots.txt Mistakes Checker
Testing Your robots.txt Configuration
Google Search Console robots.txt Tester: Navigate to Settings → robots.txt in GSC. You can see the currently active file and test any URL against it. The tester shows which rule is matching and whether the URL is blocked or allowed.
Fetch as Google: Use URL Inspection → Test Live URL to verify a previously blocked URL is now accessible after you’ve removed a disallow rule.
Third-party validators: Tools like Merkle’s robots.txt tester and ryte.com validate your syntax and flag common issues.
Manual check: Visit yourdomain.com/robots.txt in a browser. Read every line. If you haven’t read your robots.txt file in the last 3 months, do it now.
Build robots.txt review into your deployment checklist. It’s a two-minute check that has saved countless sites from catastrophic ranking losses.
Key takeaway: robots.txt is a crawl control tool, not a security mechanism. Block what wastes crawl budget; leave open everything that should be indexed. Test it after every change.
Frequently Asked Questions
Q: Does robots.txt affect ranking for pages I don’t block?
A: Indirectly, yes. A well-configured robots.txt conserves crawl budget for your valuable pages, which can improve crawl frequency and indexation speed for those pages.
Q: Can I have multiple robots.txt files for different subdomains?
A: Yes. Each subdomain (blog.example.com, shop.example.com) can have its own robots.txt at its root. They’re completely independent.
Q: Should I list my sitemap in robots.txt?
A: Yes, always. Add Sitemap: https://yourdomain.com/sitemap.xml at the bottom of your robots.txt. It’s not required, but it helps all search engines discover your sitemap automatically.
Q: What happens if I accidentally block Googlebot?
A: Your pages stop being crawled. Pages already indexed may remain for a while but will eventually drop from the index. GSC will warn you under Coverage. Fix the robots.txt and resubmit your sitemap to accelerate re-crawling.
Q: Do I need to resubmit my sitemap after changing robots.txt?
A: If you’ve unblocked pages that were previously blocked, yes — resubmit your sitemap in GSC to prompt faster re-crawling of those URLs.
Conclusion
robots.txt is deceptively simple and catastrophically powerful. One misplaced Disallow: / can drop your entire site from Google within days. One forgotten block on your JavaScript files can silently kill rendering for months.
Make robots.txt review a standard part of every site launch, migration, and major deployment. Use GSC’s tester after every change. Document every rule with comments so future developers understand the intention.
The file is 30 lines long and lives at your root domain. Given what it controls, it deserves more attention than most teams give it.
Let Ignited Nepal Handle This
robots.txt errors are silent killers — you often won’t know there’s a problem until GSC shows a Coverage collapse or traffic drops 50% overnight. Our technical SEO audits include a full robots.txt review, wildcard rule testing, and cross-validation against your sitemap and crawl logs.
→ Request a Free Technical SEO Audit
Written by the Ignited Nepal SEO team. We build organic search systems for businesses across Nepal, Australia, UAE, USA, UK, and beyond. ignitednepal.com