1
Check what you are blocking
Make sure robots.txt is not disallowing pages you want indexed. A blanket Disallow: / blocks everything.
SEO and discoverability
Robots and sitemap checker
Check a site robots.txt and find and validate its sitemap.
Free tool vs Janitor
This free tool reads robots.txt and counts the first sitemap. Janitor validates the sitemap, follows sitemap index files, confirms the URLs resolve and re-checks crawlability continuously.
See the full site-wide check in Janitor → Sample result Crawlable
moveinsights.co.uk
Crawl directives
robots.txt Found
Sitemap Found
URLs listed 248
Your result
What this checks
How search engines read your site
robots.txt tells search engines which parts of a site they may crawl, and usually points at the sitemap. The sitemap lists the URLs you want indexed.
This tool fetches robots.txt, reads the rules and any sitemap references, then fetches the sitemap and counts the URLs it lists.
Why it matters
One line can hide a whole site
A stray Disallow line in robots.txt can hide a whole site from search. A missing or stale sitemap means new pages take longer to be found.
These two files are small but they carry a lot of weight, and they are easy to break during a launch or a migration.
How to fix it
Keep crawlers pointed the right way
2
Reference the sitemap
Add a Sitemap: line to robots.txt pointing at the live sitemap URL.
3
Keep the sitemap current
Regenerate it when pages are added or removed, and make sure it returns 200, not a 404.
One of around two dozen checks
Robots and sitemap checker is one check
Janitor watches robots.txt and sitemap health automatically across every client site and puts it in a branded report you can send.
Keep reading
Related tools and guides
FAQ
Robots and sitemap checker FAQ
What does the tool check?
It fetches robots.txt, shows its content and any sitemap references, then fetches the sitemap and counts the URLs in it.
My sitemap is at a custom path. Will it be found?
If robots.txt references it with a Sitemap: line, yes. Otherwise the tool checks the conventional /sitemap.xml.
Can Janitor watch these for changes?
Yes. Janitor monitors robots.txt and the sitemap on every site and flags changes, so an accidental block does not go unnoticed.
Get started
Check it once, or watch it for every client
Janitor runs around two dozen checks on every site you manage and turns them into a branded report.
30-day free trial. No credit card required.