Loading...
Instantly fetch and validate any website's crawler directives. Identify blocking rules and sitemap locations.
Ready to analyze
Enter a URL above to fetch the robots.txt file
Specifies which crawler the rules apply to. Use * for all bots.
Blocks access to specific paths. Disallow: /admin/ blocks admin area.
Points crawlers to your XML sitemap for better indexing coverage.
Get answers to commonly asked questions