ZenovayTools

Robots.txt Generator

Generate a robots.txt file with common user-agent rules, disallow paths, crawl-delay settings, and sitemap URL. Includes presets for common CMS platforms (WordPress, Shopify, Next.js) and SEO configurations. Preview and copy the result.

Quick presets:

Disallow
None — add paths or leave empty to allow all
Allow
None — add paths or leave empty to use default

Deploy this file to: https://yourdomain.com/robots.txt

How to Use Robots.txt Generator

  1. 1Select a preset or configure custom rules for each crawler.
  2. 2Add or remove Disallow and Allow paths for each user agent.
  3. 3Set a crawl delay if needed and add your sitemap URL.
  4. 4Preview and copy the generated robots.txt content.
Zenovay

Privacy-first analytics for your website

Understand your visitors without invasive tracking. GDPR compliant, lightweight, and powerful.

Explore Zenovay

Related Tools

Meta Tag AnalyzerAnalyze meta tags of any webpage. Check title, description, Open Graph, Twitter cards, and get SEO recommendations.
Open Graph CheckerPreview how your page looks when shared on Facebook, Twitter, and LinkedIn. Check all OG and Twitter Card tags.
HTTP Header CheckerInspect HTTP response headers of any URL. Check security headers, caching, content type, and more.
Robots.txt ValidatorValidate and analyze your robots.txt file. Check rules, sitemaps, and common crawling issues.

Frequently Asked Questions

What is robots.txt and how does it work?▾
robots.txt is a text file in the root of a website that tells search engine crawlers which pages or sections they should or should not visit. It follows the Robots Exclusion Protocol (REP). Note: robots.txt is advisory — well-behaved crawlers (Googlebot, Bingbot) follow it, but malicious bots may ignore it. It is not a security mechanism and does not block access to content, it only requests that crawlers skip certain URLs.
What is the difference between Disallow and Allow?▾
Disallow tells crawlers not to access a path. Allow overrides a Disallow for a more specific path. For example: Disallow: /private/ blocks the entire /private/ directory, but Allow: /private/public.html allows that specific file. More specific rules take precedence. An empty Disallow: means allow everything (effectively a no-op). Disallow: / blocks the entire site.
How do I block a specific bot?▾
Use User-agent: BotName followed by rules for that bot. Common bot names: Googlebot (Google), Bingbot (Bing), Slurp (Yahoo), DuckDuckBot (DuckDuckGo), facebookexternalhit (Facebook). The wildcard User-agent: * applies to all bots not otherwise specified. Bot-specific rules override the wildcard for that bot. To block AI training bots: GPTBot (OpenAI), CCBot (Common Crawl), ClaudeBot (Anthropic), Google-Extended.
Should I include my sitemap in robots.txt?▾
Yes — adding Sitemap: https://example.com/sitemap.xml to robots.txt helps search engines discover your sitemap without needing to submit it manually to each search console. You can list multiple sitemaps. The Sitemap directive is not part of the original robots exclusion standard but is supported by all major search engines including Google, Bing, and Yahoo.
What is Crawl-delay and should I use it?▾
Crawl-delay specifies a minimum number of seconds between requests from the crawler. Some crawlers (Bingbot, Yandex) support it, but Google ignores it — use Google Search Console to manage Googlebot's crawl rate instead. Use Crawl-delay only if server load from crawlers is a problem, typically 1-10 seconds. High crawl delays can slow down how quickly Google discovers new or updated content.