Robots.txt Generator
Build a valid robots.txt file with allow and disallow rules, crawl-delay, and sitemap links — with a live preview you can copy or download.
Generated robots.txt
User-agent: * Allow: /
How to Use This Tool
Start from a preset — "Allow All Crawlers," "Block All Crawlers," or a blank custom configuration — then adjust the user-agent, allow paths, and disallow paths for your site. Add additional user-agent groups if you want different rules for different crawlers, and optionally list one or more sitemap URLs. The generated file updates live; copy it or download it directly as robots.txt.
How It Works
The tool assembles standard robots.txt syntax: a User-agent line followed by any number of Allow and Disallow lines, an optional Crawl-delay, repeated for each group you add, followed by any Sitemap lines at the end. Each field maps directly to its corresponding line in the output, so you can see exactly how your choices translate into the final file before downloading it.
Why This Tool Is Useful for SEO
A correctly formatted robots.txt file is one of the first things search engine crawlers check when they visit a site, and it's easy to introduce a syntax mistake by hand — a stray disallow rule can accidentally block an entire site from being crawled. Building the file through a structured form rather than typing it from memory reduces that risk and makes it easy to review before publishing.
Best Practices
Keep the default User-agent: * group unless you specifically need different rules for different crawlers. Only disallow paths you genuinely don't want crawled, such as admin areas or duplicate parameterized URLs — remember that disallowing a page doesn't remove it from search results if it's already indexed or linked elsewhere. Always include your sitemap URL so crawlers can find it easily.
Frequently Asked Questions
Does robots.txt keep a page out of Google entirely?
Not necessarily. Disallowing a URL in robots.txt stops well-behaved crawlers from requesting it, but if the URL is linked from elsewhere, it can still appear in search results without its content being crawled. Use a noindex meta tag for pages you want fully excluded.
Can I have more than one User-agent group?
Yes — click "Add Another User-agent Group" to create separate rules for different crawlers, such as one general rule and one specific to a particular bot.
Where do I upload the generated file?
Place it at the root of your domain as yoursite.com/robots.txt — most crawlers only check that exact location.