SEO and discoverability
Robots and sitemap checker
Check a site robots.txt and find and validate its sitemap.
Free tool vs Janitor
This free tool reads robots.txt and counts the first sitemap. Janitor validates the sitemap, follows sitemap index files, confirms the URLs resolve and re-checks crawlability continuously.
See the full site-wide check in Janitor
What this checks
robots.txt tells search engines which parts of a site they may crawl, and usually points at the sitemap. The sitemap lists the URLs you want indexed.
This tool fetches robots.txt, reads the rules and any sitemap references, then fetches the sitemap and counts the URLs it lists.
Why it matters
A stray Disallow line in robots.txt can hide a whole site from search. A missing or stale sitemap means new pages take longer to be found.
These two files are small but they carry a lot of weight, and they are easy to break during a launch or a migration.
How to fix common failures
Check what you are blocking
Make sure robots.txt is not disallowing pages you want indexed. A blanket Disallow: / blocks everything.
Reference the sitemap
Add a Sitemap: line to robots.txt pointing at the live sitemap URL.
Keep the sitemap current
Regenerate it when pages are added or removed, and make sure it returns 200, not a 404.
Robots and sitemap checker is one check. Janitor watches robots.txt and sitemap health automatically across every client site and puts it in a branded report.
Start your free trialKeep reading
Related
FAQ
Robots and sitemap checker FAQ
What does the tool check?
It fetches robots.txt, shows its content and any sitemap references, then fetches the sitemap and counts the URLs in it.
My sitemap is at a custom path. Will it be found?
If robots.txt references it with a Sitemap: line, yes. Otherwise the tool checks the conventional /sitemap.xml.
Can Janitor watch these for changes?
Yes. Janitor monitors robots.txt and the sitemap on every site and flags changes, so an accidental block does not go unnoticed.
Get started
Check it once, or watch it for every client
Janitor runs around two dozen checks on every site you manage and turns them into a branded report.
30-day free trial. No credit card required.