Loading...
Analyze and fetch any website's robots.txt file. Check crawler permissions and SEO directives.
The robots.txt file tells web crawlers which pages or sections of a website should not be crawled or indexed.
Common directives:
User-agent: Specifies which crawlerDisallow: Blocks access to pathsAllow: Permits access to pathsSitemap: Points to XML sitemapEnter a website URL above to fetch and analyze its robots.txt file
User-agent: * for universal rulesDisallow: paths for restricted contentSitemap: URLs for XML sitemapsGet answers to commonly asked questions