Robots.txt Generator
Generate, allow, disallow, and manage crawler rules for your website SEO.
Rules Configuration
Generated Output
What is Robots.txt?
The robots.txt file is a text file that resides in the root directory of your website. It instructs search engine crawlers (bots) which pages or files they can or cannot request from your site. It is mainly used to avoid overloading your site with requests and to keep non-public pages off Google.
This tool helps you generate the syntax correctly without needing to manually write the code, ensuring you don't accidentally block important content.
Common use cases
- Block Admin Pages: Prevent crawlers from indexing `wp-admin` or login pages.
- Guide Crawlers: Tell bots specifically where your XML Sitemap is.
- Block Private Files: Keep PDF, staging environments, or internal docs hidden.
- Crawl Delay: Slow down aggressive bots (though Google ignores this).
Frequently Asked Questions (FAQ)
1. Where should I put the robots.txt file?
It must be in the top-level root directory. E.g., `https://example.com/robots.txt`. Subdirectories won't work.
2. Does robots.txt prevent hacking?
No. It relies on the "honor system". Bad bots ignore it. Use password protection for sensitive data.
3. What does User-agent: * mean?
The asterisk (`*`) is a wildcard for "all bots". Rules under this apply to everyone unless a specific bot rule exists.
- User-agent: The name of the bot (e.g., Googlebot).
- Disallow: Paths bots should NOT visit.
- Allow: Exceptions to a Disallow rule.
- Sitemap: Absolute URL to your XML map.
Googlebot (Google Search)Bingbot (Bing Search)GPTBot (ChatGPT)CCBot (Common Crawl)Twitterbot (Social Cards)