Steven Black's unified hosts file

Discussion in 'other software & services' started by summerheat, Aug 4, 2016.

  1. summerheat

    summerheat Registered Member

    Joined:
    May 16, 2015
    Posts:
    2,199
    I just came across https://github.com/StevenBlack/hosts which offers a unified hosts file comprised of several well-known hosts files (Adaway, MVPS, Dan Pollock, Malware Domain List, Peter Lowe).

    The big advantage is that duplicates are already removed. This is not only very convenient if you're using a system-wide hosts file but also for uMatrix and uBlock0 as the download size for the updates of the various included hosts files is significantly reduced.

    Very useful service, indeed!
     
  2. Overkill

    Overkill Registered Member

    Joined:
    Mar 16, 2012
    Posts:
    2,343
    Location:
    USA
    Nice! where is the link located that I can add to hostsman?
     
  3. Nitty Kutchie

    Nitty Kutchie Registered Member

    Joined:
    Apr 10, 2015
    Posts:
    160
  4. WildByDesign

    WildByDesign Registered Member

    Joined:
    Sep 24, 2013
    Posts:
    2,587
    Location:
    Toronto, Canada
    That's a great combination of reputable hosts files, for sure. Also great for the fact that the duplicates are removed.
     
  5. Overkill

    Overkill Registered Member

    Joined:
    Mar 16, 2012
    Posts:
    2,343
    Location:
    USA
  6. summerheat

    summerheat Registered Member

    Joined:
    May 16, 2015
    Posts:
    2,199
  7. summerheat

    summerheat Registered Member

    Joined:
    May 16, 2015
    Posts:
    2,199
    This must be a Hostsman problem. The format of the hosts file is as it should be. And I don't have any problems using it in uMatrix and in Adaway (on my Android phone).
     
  8. quietman

    quietman Registered Member

    Joined:
    Dec 27, 2014
    Posts:
    511
    Location:
    Earth .... occasionally
    @summerheat

    Thank you for posting ..... I've been wanting something like this !

    I used to use Hostsman but switched to Hosts Block some time ago , and I'm not seeing any compatibility issues there.

    The removal of duplicates is definitely a big bonus , especially for uBlock0
     
  9. mapletree

    mapletree Registered Member

    Joined:
    Oct 31, 2012
    Posts:
    11
    -- IE.11 / w7x64 / 16 gb RAM / 3rd gen i7 --


    History = AdBlock plus for IE for 2 months (EasyList only) -- a better experience than before,
    but didn't seem to auto-update as it should - full permissions granted in PFW, too

    *still got pre-video YouTube ads,

    *still got new tab pop-outs whilst entering Search terms or copy-clicking on a webpage,
    in various forums

    NOW !! - Using Mr. Black's file for 2 weeks - very, VERY happy - night/day difference!

    especially the IMMEDIATE lack of ads prefacing YouTube vids!!

    I just dL'd the desired file and manually installed it,
    but I am also experimenting with every Hosts file manager I could find over the past few days --

    [so far, the most robust & feature-rich seem to be the trusty HostsMan,
    HostsBlock by Brocke here @ Wilders, HostsXpert by funkytoad,
    Host Mechanic by HackerPunk1 (askVG), and Host File Editor by weebly (mvps) --

    a couple of others show up in search, but they simply won't load up for me,
    even w/ nothing else running - while those mentioned will fire up inside 10 seconds. ]

    Cheers

    (btw - a little trick I stumbled on whilst annoyed that ABP wasn't blocking these YouTube ads --

    the 'magic number' seemed to be to allow 5 seconds of ad play, then F5 Refresh,
    and then the desired video plays -- works about 80% of the time --
    occasionally had to do this little dance twice -- still worth it if the ad > 30 seconds or so)
     
  10. chrcol

    chrcol Registered Member

    Joined:
    Apr 19, 2006
    Posts:
    982
    Location:
    UK
    one needs to be very careful with these host lists, they dont get the same care as things like easylist.

    The 2 main problems are, is there is domains in some of these lists that are nothing to do with tracking or malware, but go against the author's belief's e.g. betting sites.

    Second is some domains need partial filtering but not an outright block otherwise websites break.
     
  11. summerheat

    summerheat Registered Member

    Joined:
    May 16, 2015
    Posts:
    2,199
    The hosts files used here are of good quality. But it's true that this doesn't apply to other lists. I've used hosts files in the past that produced many false positives.

    That's why there are several variants of that list. And that's why I prefer using those lists in uMatrix as it's much easier to whitelist specific sites.

    As a matter of fact most domains need a partial filtering as blocking the whole domain is usually effectless in hosts files. Examples:

    Code:
    0.0.0.0 doubleclick.net
    does not block, e.g.,
    Code:
    ad-g.doubleclick.net
    ad.doubleclick.net
    ad.mo.doubleclick.net
    googleads.g.doubleclick.net
    In other words, through a hosts file only the specific entries therein are blocked. However, if you're using, e.g., dnsmasq as an alternative and you add
    Code:
    address=/doubleclick.net/0.0.0.0
    to its configuration file all sub-domains of doubleclick.net are also blocked.
     
  12. chrcol

    chrcol Registered Member

    Joined:
    Apr 19, 2006
    Posts:
    982
    Location:
    UK
    by partial filtering I dont mean sub domains, I mean partial filtering of http data.

    Also yes mvps and pollocks will both break some sites.
     
  13. ScriptTiger

    ScriptTiger Registered Member

    Joined:
    Mar 14, 2018
    Posts:
    2
    Location:
    Citizen of the World
    I know this is a bit old, but a lot of people have been reporting the Hostman incompatibility issue with Steven Black's hosts files. It is indeed an issue with Hostman, more specifically the HTTP(S) fetcher it uses is not compatible with the way files are served by raw.githubusercontent.com. However, Steven Black also provides the "Non Github mirror" links which are compatible. github.io domains also serve Web content the standard way, so if you're looking for CRLF hosts files or ZIP-packaged hosts files, you can also try the links here:
    https://scripttiger.github.io/hosts-packages/
    Steven Black had to drop the ZIP packages from his main repository due to the added baggage they bring with git history, but some other older applications still require it so you can still get support for those from the link above.

    Also, if your system seems to resolve DNS queries more slowly after implementing a hosts file blacklist, you can also try playing with the "compressed" format at the following link:
    https://scripttiger.github.io/alts/
    It dramatically reduces the line count of the file so it can be parsed more quickly by the DNS client cache while also keeping all of the data intact. Steven Black's main repository also offers a similar format option available by using the Python script.
     
  14. conceptualclarity

    conceptualclarity Registered Member

    Joined:
    Jun 11, 2013
    Posts:
    58
    Location:
    USA
    Support for adding Steven Black hosts is built into the application Hosts File Editor+.

    I am very wary, however, of using multiple sources for ad-blocking. False positives are inevitable with ad-blocking. You will at times be denied access to sites you have good reason to want to visit, necessitating that you undertake to whitelist the site. The more sources of ad-blocking you have, the longer it will take to hunt down the source of the problem so you can undo it.

    Some of Black's lists block so-called "fake news". No thanks. I don't need Steven Black to tell me what news sites to look at.

    I tried to use Hosts File Editor+ to add the hpHosts files EMD and FSA with the "Download and install update list (.txt)" line under Tools<Update hosts file. Turns out those were big files, 4 to 5 MB, and HFE+ didn't handle them well. Adding them through HostsMan was a lot easier, but on my old system the DNS Client Service went crazy on CPU usage and wouldn't stop till I reset my HOSTS file to default, i.e., empty.

    I wonder if you can run a bigger HOSTS file on a newer system, which I will be moving to.
     
  15. ScriptTiger

    ScriptTiger Registered Member

    Joined:
    Mar 14, 2018
    Posts:
    2
    Location:
    Citizen of the World
    This is a valid concern, especially if you're managing multiple systems. However, I will say that Steven Black's GitHub community (https://github.com/StevenBlack/hosts/issues) is quite responsive to any issues made. Having highly active curators is actually one of the goals of the project. Sometimes controversial conflicts may take a while to be discussed, such as if you're having problems due to Facebook or other social tagging/tracking sites being intentionally blocked. For the most part though the average person having a problem logging into a game server, watching a video, or other casual Web surfing activities like that are usually resolved within 24 hours with feedback from at least 2 or 3 people and a response from the upstream source marking something as a false positive and resolving the issue.

    The "fake news" piece is an optional extension and is actually managed by Mark "Marktron" Allen, not Steven Black. It was included as an option due to user demand. If you don't want to use it, you don't have to. It's not included in the base hosts file.
     
  1. This site uses cookies to help personalise content, tailor your experience and to keep you logged in if you register.
    By continuing to use this site, you are consenting to our use of cookies.