Google said that over the past 25 years Robots Exclusion Protocol (REP) protocol has been agreed is a standard, to webmaster tools crawlers personnel and developers to bring a lot of uncertainty. Google now announced that it will take the lead in working to make REP become an industry standard, as part of this effort, it is open source parser robots.txt own use, the source code is hosted on GitHub, using Apache License 2.0 license. robots.txt parser is a C ++ library for parsing and matching rules robots.txt file, which is already about 20 years old, contains the code written in the 1990s.
Manuscripts: Solidot