Is there a way to prevent our sites from being indexed/visited by robots?

Discussion in 'privacy general' started by Jim Verard, Oct 15, 2007.

Thread Status:
Not open for further replies.
  1. Jim Verard

    Jim Verard Registered Member

    Joined:
    Jun 5, 2007
    Posts:
    205
    Greetings.

    I am asking if there's a way to prevent search engines like Google, Yahoo and others from indexing our site and placing any robots from indexing and recording all our documents.

    I just found out these links:

    http://pageresource.com/zine/robotstxt.htm

    There's also this extensive FAQ:

    http://www.robotstxt.org

    The first one explains a little how to do that job:

    I make a single robots.txt file which have the lines:

    User-agent: *
    Disallow: /


    However, it's placed on a free host server. The address will be the following:

    http://myusername.myfreehostserver.com/robots.txt

    There's no way to prevent my website from being indexed from these search engines? And what about some IP Deny Manager, available sometimes on the Cpanel of your host service?

    If this option is not available from the beginning, and we have one board on our website, how did you know what IPs can be placed on your admin panel (banned IPs) to prevent the same robots from Google, and others?

    The FAQ also explained:

    My english is not so good, so I take that "parsed" means "verified".

    Do I have to place that line on every single file created on my website? Like I said I have a board like vbulettin, what lines should be included on my files (most of them are not written on HTML, they are always PHP).

    My last question should be: is there a way to place some Javascript code on all of our pages, to prevent our host service from using Google Analytics code?

    I just sign up for a host and they are using this service on every single one of my pages. There's no option to remove this service manually here, which is not related to the statistics function also available on my Cpanel.

    Any help will be appreciated. :)
     
  2. ravin

    ravin Registered Member

    Joined:
    May 2, 2003
    Posts:
    241
    Location:
    South Carolina
  3. dNor

    dNor Registered Member

    Joined:
    Oct 3, 2007
    Posts:
    212
    Location:
    Irvine, CA, USA
    You would place META tags in the header PHP section of your forum.
     
Loading...
Thread Status:
Not open for further replies.