Robots Crawlable

< More Features

Everyone wants their website to appear in the search engines. It's important to ensure your website is viewable to search engines at all times. Meta nofollow, meta noindex, robots.txt, X-ROBOTS header, ... it's a lot to keep track of. Janitor can take care of that for you.

How we check your website is crawlable by robots

We'll crawl your website, and we'll check for some key things. First and foremost, let's start with the robots entry-point for all search engine crawlers, the robots.txt.

Janitor will take a look at robots.txt and ensure your website is crawlable. Of course, there are plenty of legitimate reasons to have disallow rules within the robots.txt file, but often a single character out of place can cause the whole site to become uncrawlable.

Meta tags on each page are obeyed by search engines. Janitor will make sure your website's home page and the immediate pages off it are crawlable.

It's possible to block robots from crawling your website by using a server header called X-ROBOTS. We'll check for the header, and if we find a problem we'll let you know.

Of course, if we find any problems, we'll contact you immediately.