Robots.txt Analyzer

It fetches your robots.txt, validates the syntax, checks that declared sitemap URLs are reachable, and cross-references Disallow rules against URLs in your sitemap to detect accidental crawl blocks. It also flags sensitive paths disclosed in Disallow rules (the Streisand effect).

If your robots.txt has Disallow: /blog/ but your sitemap contains /blog/post-1, /blog/post-2, etc., Google sees those URLs in the sitemap but cannot crawl them due to the Disallow rule. This conflict causes indexing failures. The analyzer detects these mismatches and shows which specific URLs are blocked.

No. A missing robots.txt (404 response) is fine — search engines default to crawling everything. It only becomes an issue if you need to restrict specific paths. However, having a robots.txt with a Sitemap: directive helps search engines discover your sitemap faster.

"Disallow: /" blocks all crawlers from indexing any page on your site. This is the most common accidental SEO disaster — a developer accidentally pushes a staging robots.txt to production. If Googlebot respects this, the entire site can disappear from search results within days.

Listing sensitive paths in Disallow rules (like /admin, /backup, /private-api) in robots.txt makes them publicly discoverable — anyone can read your robots.txt. Attackers specifically look at robots.txt for interesting paths to probe. To restrict access to sensitive paths, use server-level authentication instead of robots.txt.

Major search engine crawlers (Googlebot, Bingbot, DuckDuckBot) respect robots.txt. However, scrapers, data brokers, AI training crawlers, and malicious bots often ignore it entirely. robots.txt is a convention, not a security mechanism — never rely on it to protect sensitive content.

A sitemap index is a root XML file that links to multiple individual sitemaps (useful for large sites with thousands of URLs). A regular sitemap XML lists page URLs directly. This tool detects which type your site uses and checks reachability for both.