Google open-sources robots.txt parser in push to make Robots Exclusion Protocol an official standard July 1, 2019 https://venturebeat.com/2019/07/01/...bots-exclusion-protocol-an-official-standard/