Google Releases a robots.txt Specification Test and a Java robots.txt Parser

Google has released two new opensource robots.txt projects

• First, they are releasing a testing framework for robots.txt parser by Andreea Dutulescu, the project provides a testing tool that can validate whether a robots.txt parser follows the Robots Exclusion Protocol, or to what extent.
• Second, we are releasing an official Java port of the C++ robots.txt parser, created by Ian Dolzhanskii.