Google open-sources robots.txt parser in push to make Robots Exclusion Protocol an official standard

Discussion in 'other software & services' started by guest, Jul 1, 2019.

  1. guest

    guest Guest

    Google open-sources robots.txt parser in push to make Robots Exclusion Protocol an official standard
    July 1, 2019
    https://venturebeat.com/2019/07/01/...bots-exclusion-protocol-an-official-standard/
     
  1. This site uses cookies to help personalise content, tailor your experience and to keep you logged in if you register.
    By continuing to use this site, you are consenting to our use of cookies.