SEOTools.info / Tools / robots.txt Checker
robots.txt Checker
Fetch and read a site's robots.txt — see its rules, disallow count and declared sitemaps.
We fetch /robots.txt from the domain.
Next step: robots.txt Generator →
About the robots.txt Checker
Fetches and reads a site's robots.txt — the file that tells search crawlers where they may and may not go. A single wrong Disallow line can hide an entire site from Google.
What this tool checks
- Shows the live robots.txt content
- Counts Disallow rules
- Lists declared sitemaps
Frequently asked questions
Does robots.txt stop indexing?
No — it stops crawling. To keep a page out of the index use a noindex meta tag instead; a blocked page can still be indexed via links.
Should I block CSS/JS?
No — Google needs them to render pages. Blocking them can hurt how your pages are understood.
Track this across every page automatically
Search Console, rank tracking and site monitoring in one panel — free account, no card.