| http://www.sxw.org.uk/computing/robots/check.html | |
|
A number of small errors in robots.txt file can lead to many problems for search engine crawlers.
This tool will check for robots.txt syntax contents for correct specifications, and will provide necessary warnings.
|