[Thread split] Hosts file et al

[Espresso]

I use my HOSTS file for ad blocking and I noticed today that MSE had quarantined the HOSTS file because of possible hijacks. After allowing it, I check the etc folder and noticed there was a copy of HOSTS made by MSE , so I compared the two and the restored version had ad.doubleclick.net removed.

[funkydude]

How is that suspicious? It would me smart for malware to modify the HOSTS file and divert a popular advertising site to a malicious one, that way they have a higher chance of infecting you further. So it could easily be a FP.

Yet another reason not to use a HOSTS file for something like ad blocking and use a real tool designed to do just that.

[Espresso]

Considering that I have ~2700 entries all linked to 0.0.0.0, I wouldn't consider the focus on one linked to 0.0.0.0 to be smart at all, and mildly suspicious at least.

[marc57]

I don't know if this works, ( I haven't had a virus in a long time) but a "Malware Expert" told me to set the HOSTS file to read only to keep malware from changing it.

[kupo]

Originally Posted by marc57

I don't know if this works, ( I haven't had a virus in a long time) but a "Malware Expert" told me to set the HOSTS file to read only to keep malware from changing it.

Nah, there's no harm in doing this but it's barely a protection.

[Hungry Man]

An attacker would need admin rights to write to the host file. If they have those rights they can already change the permissions I think.

[marc57]

Thanks for the info.

[xxJackxx]

Originally Posted by Hungry Man

An attacker would need admin rights to write to the host file. If they have those rights they can already change the permissions I think.

Yep, once you get to this point you have bigger problems.

[funkydude]

Originally Posted by Espresso

Considering that I have ~2700 entries all linked to 0.0.0.0, I wouldn't consider the focus on one linked to 0.0.0.0 to be smart at all, and mildly suspicious at least.

Suspicious? Right, because Microsoft has an interested in unblocking Google owned advertising services from your machine, right...?

What does the amount of entries have to do with anything? Malware modifies/adds specific entries. Like I've already told you, it's quite possible that malware exists that hijacks that specific entry, which set off MSE. Ofcourse MSE finds it suspicious, like I've already stated, using a HOSTS file as an ad blocker is silly.

Also, using 0.0.0.0 has already been proven flawed and can cause more issues than it solves based on what programs you're using.

Myth: there is no known documentation that proves a different prefix is faster than "127.0.0.1" ... yes "0.0.0.0" is a valid alternative for users that are running a Web Server. Over the years some people try to prove that (example - 255.255.255.255) one prefix if faster ... but they fail to see the big picture ... most major security products have the ability to scan the HOSTS file and they accept the prefix "127.0.0.1". What happens when one of these products has a fit when it encounters an unknown prefix? It's safer to stay with the "industry standard" of 127.0.0.1

http://winhelp2002.mvps.org/hostsfaq.htm

Looks like there's your explanation of why MSE was set off.

[Hungry Man]

The only thing it breaks is the assumption that 127.0.0.1 is being used. If you use 0 it saves space.

[funkydude]

Originally Posted by Hungry Man

The only thing it breaks is the assumption that 127.0.0.1 is being used.

Being the standard loopback interface it's a pretty safe assumption, going against the standards is what breaks it.

Originally Posted by Hungry Man

If you use 0 it saves space.

That adds no benefit whatsoever, especially not for saving 2 digits... cmon. What you want is to reduce line count, not line length. That's why optimizers will make lines longer to reduce line count.

[Hungry Man]

Being the standard loopback interface it's a pretty safe assumption, going against the standards is what breaks it.

I'm just saying it's not like the OS cares the problem is that products like Spybot can't handle it.

That adds no benefit whatsoever, especially not for saving 2 digits... cmon. What you want is to reduce line count, not line length. That's why optimizers will make lines longer to reduce line count.

I don't think it works on earlier Windows but you can use just "0" and over 2700 lines you save a bit of space. It's all held in RAM so access time/ read won't be an issue it's more about saving yourself however much RAM that saves.

[xxJackxx]

Originally Posted by Hungry Man

The only thing it breaks is the assumption that 127.0.0.1 is being used. If you use 0 it saves space.

Yes, it is also faster. 127.0.0.1 is a valid address on ANY machine, web server or no. 0.0.0.0 will always be faster as something checking 127.0.0.1 will wait for a response. I have been using 0.0.0.0 for 10 years and have never had an issue because of it.

[funkydude]

Originally Posted by xxJackxx

Yes, it is also faster. 127.0.0.1 is a valid address on ANY machine, web server or no. 0.0.0.0 will always be faster as something checking 127.0.0.1 will wait for a response. I have been using 0.0.0.0 for 10 years and have never had an issue because of it.

Please try your best to read previous posts before posting false information, it will save you from unnecessary embarrassment.

[Cudni]

thread split from
http://www.wilderssecurity.com/showthread.php?t=322765

[Espresso]

Originally Posted by funkydude

Suspicious? Right, because Microsoft has an interested in unblocking Google owned advertising services from your machine, right...?

What does the amount of entries have to do with anything?

Because I have quite a few "popular" ad sites there, all of which were left alone.

Malware modifies/adds specific entries. Like I've already told you, it's quite possible that malware exists that hijacks that specific entry, which set off MSE. Ofcourse MSE finds it suspicious,

Specious reasoning. A link to 0.0.0.0 should be considered benign.

like I've already stated, using a HOSTS file as an ad blocker is silly.

Not silly at all. No extra processes, no local proxies required. It works.

Also, using 0.0.0.0 has already been proven flawed and can cause more issues than it solves based on what programs you're using.
http://winhelp2002.mvps.org/hostsfaq.htm

Looks like there's your explanation of why MSE was set off.

Hardly. It missed another ~2700 entries linked to 0.0.0.0.

[Espresso]

Originally Posted by funkydude

Please try your best to read previous posts before posting false information, it will save you from unnecessary embarrassment.

False? Quoting a FAQ entry by a guy who claims he can "see no noticeable difference" hardly qualifies as proof.

Here's a small test by someone who at least did repeated, timed tests:

http://hype-free.blogspot.com/2009/0...1-vs-0000.html

•Firefox (3.5) showed no difference: the timing always was a couple of milliseconds
•Internet Explorer (8) and Google Chrome (2.0) showed quite a large difference in favor of 0.0.0.0: when using it, the pages constantly loaded in tens of milliseconds (between 0 and 30), while using 127.0.0.1 meant a page load time closer to a second (between 900 and 1500 milliseconds).
•Using Opera (9.64) the results were even further apart: the 0.0.0.0 case took ~1 second, while the 127.0.0.1 case took around 25 (!!!) seconds.

YMMV.

[funkydude]

Originally Posted by Espresso

Specious reasoning. A link to 0.0.0.0 should be considered benign.

Not when it's completely against standards.

Originally Posted by Espresso

Not silly at all. No extra processes, no local proxies required. It works.

It is extremely silly, it's a flawed way of blocking because not only can it not block 100%, there is also no way to fix the high amount of false positives that comes with blanket banning thousands of domains.

Originally Posted by Espresso

Hardly. It missed another ~2700 entries linked to 0.0.0.0.

Here I thought I wouldn't need to explain something so basic, I guess not. It's not the entry that sets it off, it's the combination of the entry and the unusual IP address. For the third (can we make this the last?) time, it's possible that specific malware hijacks that entry which sets it off as suspicious.

False? Quoting a FAQ entry by a guy who claims he can "see no noticeable difference" hardly qualifies as proof.

LOL! "A guy who claims"? This "guy" was probably the first ever person to publish a publicly available HOSTS file. He's literally been doing it for a decade.

But I guess you'd rather take that 3 year old blog.... with ancient browsers as reference, right? The fact will always remain, programs designed with the specific goal of blocking ads will always be superior, and faster.

[kupo]

While I believe that hosts file as adblocker is not a good idea (there are much better tools for this) and as malware blocklist as a really stupid idea (websites with exploits come and go, and you could have a lot of dead entries in your hosts file), there are good reasons to use an invalid address instead of the loopback address in hosts file. (0.0.0.0 or 255.255.255.0 instead of 127.0.0.1).
-http://hackademix.net/2009/07/01/abe-warnings-everywhere-omg/

[funkydude]

Did you read your own link in comparison to the link I posted?

Your link:

This is not very smart, especially if you installed a web server on the loopback interface (like many web developers do), because you’re spamming it with dummy requests whenever you browse an ad-laden web site.

My link:

Myth: there is no known documentation that proves a different prefix is faster than "127.0.0.1" ... yes "0.0.0.0" is a valid alternative for users that are running a Web Server. Over the years some people try to prove that (example - 255.255.255.255) one prefix if faster ... but they fail to see the big picture ... most major security products have the ability to scan the HOSTS file and they accept the prefix "127.0.0.1". What happens when one of these products has a fit when it encounters an unknown prefix? It's safer to stay with the "industry standard" of 127.0.0.1

[Espresso]

Originally Posted by funkydude

It is extremely silly, it's a flawed way of blocking because not only can it not block 100%, there is also no way to fix the high amount of false positives that comes with blanket banning thousands of domains.

I don't know anything that blocks 100%, nor do I care. There are probably ~50 sites that serve most of the annoying ads and as long as I get those I'm happy. It's not that big a deal. There are hardly any "false positives" from blocking an advertising domain, only if a company links content delivery with an ad server which is rare and an easy fix. In my experience it was more common to get false positives from ad blockers that block by parsing urls and image sizes.

Here I thought I wouldn't need to explain something so basic, I guess not. It's not the entry that sets it off, it's the combination of the entry and the unusual IP address. For the third (can we make this the last?) time, it's possible that specific malware hijacks that entry which sets it off as suspicious.

Using 0.0.0.0 in a HOSTS file is not that unusual (the practice has been around since the 90s) and the link is removed if I use 127.0.0.1 as well (certainly not unusual). Yes malware have been known to redirect a number of ad sites, doubleclick included, but it's still mildly suspicious that MSE is removing only the doubleclick (a MS advertising partner in Silverlight) redirect to a localhost address. MSE is being dumb or devious.

LOL! "A guy who claims"? This "guy" was probably the first ever person to publish a publicly available HOSTS file. He's literally been doing it for a decade.

Appeal to authority. He's done no timed tests so it's merely anecdotal.

But I guess you'd rather take that 3 year old blog.... with ancient browsers as reference, right?

Show me how/why new browsers would be different.

The fact will always remain, programs designed with the specific goal of blocking ads will always be superior, and faster.

I don't see how they're faster. I assume you have some properly designed tests to back up your assertion.

[funkydude]

Originally Posted by Espresso

I don't know anything that blocks 100%, nor do I care. There are probably ~50 sites that serve most of the annoying ads and as long as I get those I'm happy. It's not that big a deal.

Every single program including ABP that's designed to block ads can block them 100%, easily. That's what they are designed to do.

Originally Posted by Espresso

There are hardly any "false positives" from blocking an advertising domain, only if a company links content delivery with an ad server which is rare and an easy fix. In my experience it was more common to get false positives from ad blockers that block by parsing urls and image sizes.

You need only look for a second at the amount of specific allow rules that are needed in EasyList to see what kind of breakage outright blocking can do.

http://easylist-msie.adblockplus.org...y+easylist.tpl

I bet your "great" HOSTS file blocks doubleclick, right? Look at the amount of allow rules needed to prevent breakage there.

+d doubleclick.net /ad/can/chow/
# @@||doubleclick.net/ad/can/tvcom/$object-subrequest
# @@||doubleclick.net/adi/*.mlb/photos;*;sz=300x250;$subdocument
+d doubleclick.net /adi/amzn.*;ri=digital-music-track;
# @@||doubleclick.net/adi/dhd/homepage;sz=728x90;*;pos=top;$subdocument
# @@||doubleclick.net/adi/sony.oz.opus/*;pos=bottom;$subdocument
# @@||doubleclick.net/adi/yesnetwork.com/media;$subdocument
# @@||doubleclick.net/adj/cm.tim/*;cmpos=subscribetop;$script
# @@||doubleclick.net/adj/cm.tim/*;cmpos=upgradetop;$script
+d doubleclick.net /adj/ctv.muchmusicblog.com/*.js
# @@||doubleclick.net/adj/ftcom.*;sz=1x1;*;pos=refresh;$script
# @@||doubleclick.net/adj/gamesco.socialgaming/$script
# @@||doubleclick.net/adj/iblocal.mediageneral.wncn/*;pos=1;sz=253x300;$script
+d doubleclick.net /adj/imdb2.consumer.video/*;sz=320x240,*.js
# @@||doubleclick.net/adj/kval/health;pos=gallerytop;sz=$script
+d doubleclick.net /adj/nbcu.nbc/videoplayer-*.js
+d doubleclick.net /adj/pong.all/*;dcopt=ist;*.js
# @@||doubleclick.net/adj/profootballreference.fsv/$script
# @@||doubleclick.net/adj/wiredcom.dart/*;sz=300x250;*;kw=top;$script
# @@||doubleclick.net/pfadx/*/cbs/$object-subrequest
+d doubleclick.net /pfadx/umg.*;sz=10x*.js
+d doubleclick.net /adj/wwe.shows/ecw_ecwreplay;*;sz=624x325;*.js
# @@||doubleclick.net^*/ndm.tcm/video;$script
# @@||doubleclick.net^*/videoplayer*=worldnow$subdocument

Originally Posted by Espresso

Using 0.0.0.0 in a HOSTS file is not that unusual (the practice has been around since the 90s) and the link is removed if I use 127.0.0.1 as well (certainly not unusual). Yes malware have been known to redirect a number of ad sites, doubleclick included, but it's still mildly suspicious that MSE is removing only the doubleclick (a MS advertising partner in Silverlight) redirect to a localhost address. MSE is being dumb or devious.

It is very unusual when you consider what the HOSTS file is for, and how you're abusing it to accomplish a goal it was never intended for.

Calling MSE dumb because you can't see the simple truth of this is hilarious at best. Though I haven't quite laughed as hard as I did to your comment of it being suspicious, damn Microsoft trying to make Google more money by unblocking ad sites!! That really takes some imagination.

Originally Posted by Espresso

Appeal to authority. He's done no timed tests so it's merely anecdotal.

Feel free to provide evidence since you can clearly read this persons mind, amazing skills.

Originally Posted by Espresso

Show me how/why new browsers would be different.

That's like saying show me how/why new browsers would be faster at all, maybe it's magic!

They should be faster because they shouldn't be waiting for a timeout anymore. You can test this in IE9 for example, it most certainly does not take a second to reach a diverted host.

Originally Posted by Espresso

I don't see how they're faster. I assume you have some properly designed tests to back up your assertion.

They are designed with that specific goal in mind, and in some cases, even take advantage of Windows API such as the Filtering Platform(afaik TPLs do). Not only are they factually faster, they are logically faster.

Feel free to disprove otherwise...

At the end of the day using a HOSTS file will always be sub-par to using a dedicated ad blocker, that's a fact that cannot be denied by any knowledgeable person.

[Espresso]

Originally Posted by funkydude

Every single program including ABP that's designed to block ads can block them 100%, easily. That's what they are designed to do.

I've used them and they don't block 100% of ads. That's why their lists are constantly being upgraded.

You need only look for a second at the amount of specific allow rules that are needed in EasyList to see what kind of breakage outright blocking can do.

Isn't it amazing that I block it outright and I have no doubleclick related problems at all ?

Calling MSE dumb because you can't see the simple truth of this is hilarious at best.

It's dumb because it doesn't consider the linked IPs 0.0.0.0 and 127.0.0.1 which are benign, period.

Though I haven't quite laughed as hard as I did to your comment of it being suspicious, damn Microsoft trying to make Google more money by unblocking ad sites!! That really takes some imagination.

The sucess of the struggling Silverlight will depend on ad links from major ad services being functional. The idea that MS would like to facilitate thier success isn't far fetched at all. Like I said, it's mildly suspicious.

Feel free to provide evidence since you can clearly read this persons mind, amazing skills.

He probably would've posted them. In any case, you can search around and find tests like the one I linked that contradict his "findings".

They are designed with that specific goal in mind, and in some cases, even take advantage of Windows API such as the Filtering Platform(afaik TPLs do). Not only are they factually faster, they are logically faster.

Feel free to disprove otherwise...

The onus is on you to prove your assertion. Considering that a link to 0.0.0.0 is terminated almost instantly, I can't see any way an ad blocker could be appreciably faster.

[funkydude]

Originally Posted by Espresso

I've used them and they don't block 100% of ads. That's why their lists are constantly being upgraded.

Their lists are more often or not "constantly" updated to add allow rules, but programs like ABP can easily block 100%, they have the potential to do that, HOSTS files do not. That should be pretty easy to see.

Originally Posted by Espresso

Isn't it amazing that I block it outright and I have no doubleclick related problems at all ?

Oh, right, you count for the entire world now yeah? You're trying to disprove fact based solely on your own tiny experience? LOL.

Originally Posted by Espresso

It's dumb because it doesn't consider the linked IPs 0.0.0.0 and 127.0.0.1 which are benign, period.

Doesn't matter how benign they are, they aren't supposed to be used in a HOSTS file, that isn't the purpose of a HOSTS file.

Originally Posted by Espresso

The sucess of the struggling Silverlight will depend on ad links from major ad services being functional. The idea that MS would like to facilitate thier success isn't far fetched at all. Like I said, it's mildly suspicious.

Wait, what!? That's one hell of a stretch from ad blocking to Silverlight, which has pretty much been more or less killed off by MS anyway. I fail to see what Silverlight has to do with Google advertising, at all. Other than your desperation to justify your flawed responses, ofcourse.

Originally Posted by Espresso

He probably would've posted them. In any case, you can search around and find tests like the one I linked that contradict his "findings".

You can search around to find tests that contradict the findings contradicting the findings.

Originally Posted by Espresso

The onus is on you to prove your assertion. Considering that a link to 0.0.0.0 is terminated almost instantly, I can't see any way an ad blocker could be appreciably faster.

I'm sorry? Why is it on me to prove yet you have a free ride with your claims which are nothing other than blind faith? They aren't even logically sound. It doesn't matter what it points to, it's a fact that your system will become slower and slower the bigger the file is, why? Again, it's not designed for ad blocking.

Real ad blockers can use lists in a way that reduces the impact of the size of said list. They can also use freely available system resources specifically designed for preventing connections, such as WFP, which work at a higher level and are therefor faster than waiting for the HOSTS file. I'm not sure where the difficulty in understanding this lies.

edit: Also on another note, you cannot block IP addresses with a HOSTS file, advertising and tracking using IP addresses will pass right through.

[safeguy]

Blocking ads always involve the possibility of causing some sites to 'break'.
When there are too many entries involved, using a dedicated ad-blocker (which is specifically 'designed' for the task) has advantages over using a hosts file seeing that it may have the flexibility of a white-list system allowing entries to counter the initial block rule(s) and for most, the lists are automatically updated. This means less work for you and chances of 'false positives' are lower. This is good enough a reason to recommend dedicated ad-blocker over a hosts file to pseudo-blockads.

To be fair, however, using a hosts file to pseudo-block certain ads also has it's own advantages. Since it is included in the system, it requires no upgrades (unlike 3rd-party software) and that it is free of costs compared to a 3rd-party commercial ad-blocker (not counting IE 's TPL and ad-pseudo-blockextensions for other browsers). The pseudo-blocking implemented is also system-wide and works across multiple web browsers(not counting AdMuncher, AdFender, Privoxy, Proxomitron, etc).

While it is arguably less effective on a web browser compared to dedicated tools, using a hosts file to pseudo-block ads is not that bad an idea if one uses it to mainly pseudo-block the major advertising delivery sites (and do not mind the possible resulting breakage of certain sites). My personal suggestion is to not dump too many entries to reduce the likelihood of sites breakage.

Now, as to whether one should use 127.0.0.1 (localhost loopback address) or 0.0.0.0 or even 0 (invalid address/destination), that is a debate not worth going for. It is pretty much a personal choice.

Using the former in conjunction with eDexter (a local-only personal image web server) prevents browser errors (HTTP 404" error messages) and replaces annoying ads from filtered sites with GIF images.

Using the latter helps to reduce the size of the hosts file (and probably memory usage by a small margin) and speeds up the browsing speed as the system/browser immediately rejects the endpoint rather than to wait for a timeout (this depends on which OS and browser are in use although modern browsers may be better at this). 0.0.0.0 is also recommended if you're running web server software.

Note: If you use 0.0.0.0, you may want to add this entry just before your first 0.0.0.0 "blocking" entry:

# Special Entries
0.0.0.0 0.0.0.0 # fix for traceroute and netstat display anomaly

Without that entry, some network status and diagnostic apps will use the hostname associated with your first 0.0.0.0 "blocking" entry as the name of the default IP address.

I got that from here: http://www.dslreports.com/forum/r24622031-

Just take note that the use of 0 in the host file may not be supported on certain OS.

I personally find the argument of going with "industry standards" as quoted in the FAQ from MVPS HOSTS site, not that convincing. If we were to go by that argument, then the "industry" itself has contradicted the "standards" as the Hosts file was never initially intended for malware sites or ad blocking in the 1st place. It has a different purpose as stated here:
-http://en.wikipedia.org/wiki/Hosts_(file)#Purpose-
In short, it provide host name to IP address translation.

If security programs scan the HOSTS file and they only accept the IP address "127.0.0.1", it means the security program do not understand the hosts file intended purpose. It's time for them to change and learn to accept other IP addresses.

P.S. Personally, I do not encourage the Hosts file to be used as a mean to pseudo-block malware sites (that's another topic) and while I do find using a dedicated ad-blocker more suitable for the task of blocking ads, I am not against using hosts file to do so.