Adblocking with unbound

Discussion in 'all things UNIX' started by summerheat, Jul 21, 2018.

  1. summerheat

    summerheat Registered Member

    Joined:
    May 16, 2015
    Posts:
    2,199
    I've been using unbound as my local DNS server for a long time. @Stefan Froberg once wrote a nice documentation which explains the rationale and the basics.

    He also mentions how to use unbound for adblocking by using

    https://pgl.yoyo.org/adservers/serv...&showintro=0&useip=0.0.0.0&mimetype=plaintext

    which is the well-known hosts file maintained by Peter Lowe in the unbound format. This works well - but it's a realtively small hosts file. I prefer the one from Steven Black (several variants are available) which is a high quality hosts file that blocks way more than Peter Lowe's.

    It is not available in the unbound format but it's easy to convert it (or any other hosts file) with a simple script:

    Code:
    /usr/bin/wget -N https://raw.githubusercontent.com/StevenBlack/hosts/master/hosts
    cat hosts | grep '^0\.0\.0\.0' | awk '{print "local-zone: \""$2"\" redirect\nlocal-data: \""$2" A 0.0.0.0\""}' > /etc/unbound/local.d/ads.conf
    As all hosts files can produce false positives at times, you can whitelist specific hosts by adding a file whitelist (without the .conf extension) which contains, e.g.

    Code:
    "buch.de.
    "awin1.com.
    "uimserv.net.
    "wa.ui-portal.de.
    "tifbs.net.
    Note that every line must begin with a quotation mark and end with a dot.

    There's a second method which I actually prefer, namely creating a blocklist which contains entries that look like this:
    Code:
    local-zone: "000.0x1f4b0.com" refuse
    local-zone: "000free.us" refuse
    local-zone: "000.gaysexe.free.fr" refuse
    local-zone: "000owamail0.000webhostapp.com" refuse
    local-zone: "000tristanprod.free.fr" refuse
    local-zone: "001.0x1f4b0.com" refuse
    I prefer that method as it's easy now to filter those requests in the log file which are blocked by unbound with
    Code:
    tail -f /etc/unbound/log/unbound.log | grep "REFUSED"
    I was not able to find a way to filter only the blocked hosts by using the first method above.

    That second method has one drawback, though: The whitelist file doesn't work for whatever reason. Hence, you have to remove the hosts contained in your whitelist from the hosts file by using the comm command. This is how I do it:

    Code:
    /usr/bin/wget -N https://raw.githubusercontent.com/StevenBlack/hosts/master/hosts
    
    cat hosts | grep '^0\.0\.0\.0' | sort | awk '{print $2}' > block
    
    comm -23 block /home/heat/whitelist > block1
    
    awk '{print "local-zone: \""$1"\" refuse"}' block1 > /etc/unbound/local.d/blocklist.conf
    
    Any suggestions how to improve this are welcome.
     
    Last edited: Jul 22, 2018
  2. summerheat

    summerheat Registered Member

    Joined:
    May 16, 2015
    Posts:
    2,199
    FWIW, I've added this big blacklist which is maintained by Frank Denis, the author of dnscrypt-proxy. After some cleanup I merge both files and sort and de-duplicate the result with sort -u.

    Code:
    /usr/bin/wget -N https://raw.githubusercontent.com/StevenBlack/hosts/master/hosts
    /usr/bin/wget -N https://download.dnscrypt.info/blacklists/domains/mybase.txt
    cat hosts | grep '^0\.0\.0\.0' | awk '{print $2}' > block
    sed '/#/d; /*/d; /^$/d; /^\./d' mybase.txt > mybase
    cat block mybase | sort -u > merged
    comm -23 merged /home/heat/whitelist > merged_corr
    awk '{print "local-zone: \""$1"\" refuse"}' merged_corr > /etc/unbound/local.d/blocklist.conf
    Needless to say that I created a timer which executes above code shortly after every boot to keep those blacklists up-to-date.
     
  3. Stefan Froberg

    Stefan Froberg Registered Member

    Joined:
    Jul 30, 2014
    Posts:
    747
    You are right, it's much better to use "refuse" than "redirect" in the blocklist.
    Besides enabling easy logging it reduces the size of blocklist to half.:)
    And I like the bigger Steven Black list too
    :thumb:
     
  4. __Nikopol

    __Nikopol Registered Member

    Joined:
    Aug 13, 2008
    Posts:
    630
    Location:
    Germany
    ooh, I was looking at unbound today because it uses DNSCrypt-proxy. But the Windows version is more or less a joke. I couldn't get it to run.
    For Windows it's better to use Simple DNSCrypt. You can choose any filtering and non-filtering DNS you want with it. But ditch the GUI and do things manually for a while or you will have problems later when the GUI bugs out. xD
     
  5. reasonablePrivacy

    reasonablePrivacy Registered Member

    Joined:
    Oct 7, 2017
    Posts:
    2,002
    Location:
    Member state of European Union
    I managed it to run on my previous installation of Windows.
    It does not need to use DNSCrypt. It can use any DNS server - installed locally or remote like Cloudflare's servers.
     
  6. __Nikopol

    __Nikopol Registered Member

    Joined:
    Aug 13, 2008
    Posts:
    630
    Location:
    Germany
    DNSCrypt-proxy is not a service from anyone, it's a program that runs a proxy server on your computer that encrypts DNS requests coming over loopback. (Loopback is then configured as DNS in your ethernet interface settings) You can feed it with any DNS-provider you want.
     
  7. reasonablePrivacy

    reasonablePrivacy Registered Member

    Joined:
    Oct 7, 2017
    Posts:
    2,002
    Location:
    Member state of European Union
    Yes, I know. I will provide config for Windows 10 in the coming days.
     
  8. XIII

    XIII Registered Member

    Joined:
    Jan 12, 2009
    Posts:
    1,383
    Do you perhaps mean "can use"?

    I use unbound on my ASUS router, but don't use DNSCrypt (though it's available), since I use (encrypted) DNS over TLS (offered by Quad9).
     
  9. __Nikopol

    __Nikopol Registered Member

    Joined:
    Aug 13, 2008
    Posts:
    630
    Location:
    Germany
    Sorry I misunderstood :)
    Oh, ok. :thumb:
     
  10. Stefan Froberg

    Stefan Froberg Registered Member

    Joined:
    Jul 30, 2014
    Posts:
    747
    Hmmm...
    It been long time from my little DNS tutorial and the current unbound version is now
    1.7.3

    Quick glance shows at least following interesting new features from 1.6.7 days:

    1. Use accept4 to speed up incoming TCP (and TLS) connections, available on Linux, FreeBSD and OpenBSD.
    2. tls-win-cert option that adds the system certificate store for authenticating DNS-over-TLS connections. It can be used instead of the tls-cert-bundle option, or with it to add certificates.
    3. Can set tls authentication with forward-addr: IP#tls.auth.name And put the public cert bundle in tls-cert-bundle: "ca-bundle.pem". such as forward-addr: 9.9.9.9@853#dns.quad9.net or 1.1.1.1@853#cloudflare-dns.com
    4. Add --with-libhiredis, unbound support for a new cachedb backend that uses a Redis server as the storage. This implementation depends on the hiredis client library (https://redislabs.com/lp/hiredis/). And unbound should be built with both --enable-cachedb and --with-libhiredis[=PATH] (where $PATH/include/hiredis/hiredis.h should exist). Patch from Jinmei Tatuya (Infoblox).
    So it seems that unbound finally supports doing cert checking and for performance it now has
    accept4 and redis (in-memory but persistent on disk database) support.

    About redis and benchmarking:
    https://redis.io/topics/faq
    https://redis.io/topics/benchmarks

    And this fellow shows howto do the cert checking right with unbound
    https://www.ctrl.blog/entry/unbound-tls-forwarding

    Also from above link:
    "I also wrote about randomizing DNS forwarding servers with Knot Resolver a few weeks back. This ensures you’re not sending all your DNS traffic to one provider; making it more difficult for any one provider to build a complete profile on your online activities and behavior. Unbound will do this by default with no additional configuration assuming each of your configured DNS forwarders respond within 400 milliseconds. Note that you’re not limited to just four DNS forwarders as you are with Knot Resolver. You can configure as many DNS forwarders as you want with Unbound and it will spread your forwarding requests out among each of them automatically."

    That's really nice. :)
    Untill now, I tought that if I wanted to do loadbalancing or fail-safe, I had to add the extra nameservers to /etc/resolv.conf but I can instead add them to forward-zone and just keep 127.0.0.1 in /etc/resolv.conf
     
  11. XIII

    XIII Registered Member

    Joined:
    Jan 12, 2009
    Posts:
    1,383
    How can I find out whether my platform supports "accept4"? Where can I read more about this?

    (I use unbound 1.7.1-1 on an ASUS router running Asuswrt-Merlin 384.6-beta1, which uses the 4.1.27 Linux kernel)
     
  12. Stefan Froberg

    Stefan Froberg Registered Member

    Joined:
    Jul 30, 2014
    Posts:
    747
    Mention of it can be found https://www.unbound.net/download.html under 1.7.2 features.
    accept4 is a non-standard extension that should be automatically available and used if you have unbound 1.7.2 and Linux, FreeBSD or OpenBSD
    so as long as those requirements are fullfilled it should be automatically on.

    It can be found from unbound 1.7.2 source code under util/netevent.c file line 771
    ( wget https://www.unbound.net/downloads/unbound-1.7.2.tar.gz )

    And here is more information about the accept4 (and standard accept) system calls
    https://linux.die.net/man/2/accept4

    unbound author has also made lil guide howto optimize performance from default settings
    https://www.unbound.net/documentation/howto_optimise.html
     
    Last edited by a moderator: Jul 24, 2018
  13. Stefan Froberg

    Stefan Froberg Registered Member

    Joined:
    Jul 30, 2014
    Posts:
    747
    My new, tweaked unbound client side config for 8 CPU machine:

    num-threads: 8
    so-rcvbuf: 4m
    so-sndbuf: 4m
    so-reuseport: yes
    msg-cache-slabs: 8
    rrset-cache-slabs: 8
    infra-cache-slabs: 8
    key-cache-slabs: 8
    outgoing-range: 8192
    num-queries-per-thread: 4096
    msg-cache-size: 64m
    rrset-cache-size: 128m

    Also enabled prefetching, might improve cache hits.

    prefetch: yes
    prefetch-key: yes
     
  14. summerheat

    summerheat Registered Member

    Joined:
    May 16, 2015
    Posts:
    2,199
    Ah, that's good to know! So it doesn't have to be explicitly enabled somehow.

    Regarding cert checking in your previous post: If I add

    Code:
    forward-tls-upstream: yes
    tls-cert-bundle: /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem
    to unbound.conf I'm getting an error when restarting unbound. It seems that the first line already causes that error:
    I'm using v. 1.7.3 on Fedora so this should definitely be supported. Strange ....
     
  15. XIII

    XIII Registered Member

    Joined:
    Jan 12, 2009
    Posts:
    1,383
    Ah, than the version I get from Entware is too old for this anyway.

    Thanks!
     
  16. summerheat

    summerheat Registered Member

    Joined:
    May 16, 2015
    Posts:
    2,199
    I've just found this site which is a curated repo maintaining a whitelist for the Pi-Hole project. I think using this whitelist makes sense in order to avoid false positives as hosts files often tend to be overzealous. Quote:
    So I modified my little script accordingly:
    Code:
    /usr/bin/wget -N https://raw.githubusercontent.com/StevenBlack/hosts/master/hosts
    /usr/bin/wget -N https://download.dnscrypt.info/blacklists/domains/mybase.txt
    /usr/bin/wget -N https://raw.githubusercontent.com/anudeepND/whitelist/master/domains/whitelist.txt
    cat hosts | grep '^0\.0\.0\.0' | awk '{print $2}' > block
    sed '/#/d; /*/d; /^$/d; /^\./d' mybase.txt > mybase
    cat block mybase | sort -u > merged
    cat whitelist.txt /home/heat/mywhitelist | sort -u > whitelist
    comm -23 merged whitelist > merged_corr
    awk '{print "local-zone: \""$1"\" refuse"}' merged_corr > /etc/unbound/local.d/blocklist.conf
     
  17. Stefan Froberg

    Stefan Froberg Registered Member

    Joined:
    Jul 30, 2014
    Posts:
    747
    Are you sure that the tls-cert-bundle is under server: block and not accidentally under some other (like remote-control: or forward-zone: block) ?

    I think there is an error in that https://www.ctrl.blog/entry/unbound-tls-forwarding writing:
    forward-tls-upstream should be under forward-zone: block and not server: block I think ...

    Im currently testing 1.7.1 version with below client side config (encryption, DNSSEC and TLS authentication all enabled) and it works.

    In this conf, I have disabled UDP and IPv6.
    UDP disabled because I think maybe TCP is better delivering large stuff like DNSSEC because UDP fragmentation reason (better speed? have to test & also ask author opinion) ....
    So I only allow TCP 853 outgoing in my firewall and have options use-vc in my /etc/resolv.conf

    And IPv6 disabled because my dang ISP does not support it and Im too lazy right now to setup SSH or VPN tunneling for DNS testing....

    Next step...test 1.7.3

    #
    # Example configuration file.
    server:
    verbosity: 1
    num-threads: 8
    interface: 127.0.0.1
    port: 53
    prefer-ip6: no
    outgoing-range: 8192
    so-rcvbuf: 4m
    so-sndbuf: 4m
    so-reuseport: yes
    msg-cache-size: 64m
    msg-cache-slabs: 8
    num-queries-per-thread: 4096
    rrset-cache-size: 128m
    rrset-cache-slabs: 8
    infra-cache-slabs: 8
    do-ip4: yes
    do-ip6: no
    do-udp: no
    do-tcp: yes
    do-daemonize: yes
    access-control: 127.0.0.0/8 allow
    directory: "/etc/unbound"
    logfile: "unbound.log"
    log-queries: yes
    log-replies: yes
    pidfile: "/var/run/unbound.pid"
    hide-identity: yes
    hide-version: yes
    harden-short-bufsize: yes
    harden-large-queries: yes
    harden-glue: yes
    harden-dnssec-stripped: yes
    harden-below-nxdomain: yes
    harden-algo-downgrade: yes
    qname-minimisation: yes
    prefetch: yes
    prefetch-key: yes
    rrset-roundrobin: yes
    minimal-responses: yes
    auto-trust-anchor-file: "/etc/unbound/root.key"
    key-cache-slabs: 8
    tls-upstream: yes
    include: /etc/unbound/adservers
    tls-cert-bundle: /etc/ssl/certs/ca-certificates.crt

    forward-zone:
    name: "."
    forward-addr: <my own VPS IP, running unbound server side>@853
    forward-tls-upstream: yes
    # Quad9
    forward-addr: 9.9.9.9@853#dns.quad9.net
    forward-addr: 149.112.112.112@853#dns.quad9.net
    # Cloudflare DNS
    forward-addr: 1.1.1.1@853#cloudflare-dns.com
    forward-addr: 1.0.0.1@853#cloudflare-dns.com
     
    Last edited: Jul 24, 2018
  18. summerheat

    summerheat Registered Member

    Joined:
    May 16, 2015
    Posts:
    2,199
    Yes, indeed, you're right! I had placed both lines in the forward-zone block - and now the second line caused the error. After placing it in the server section the error is gone :thumb:
     
  19. summerheat

    summerheat Registered Member

    Joined:
    May 16, 2015
    Posts:
    2,199
    Thanks, I have very similar settings. Additionally, I've sandboxed unbound with Firejail.
     
  20. Stefan Froberg

    Stefan Froberg Registered Member

    Joined:
    Jul 30, 2014
    Posts:
    747
    Welcome :)

    I have now tested DNS latencies between my unbound client and server, both running 1.7.3 version, encryption, DNSSEC and TLS authentication and all the previously mentioned tweaking
    and got the following results (considering my average 4G latency, 30 milliseconds, is subtracted from these values):

    min latency seen: 269 milliseconds
    max latency seen: 799 milliseconds

    So in worst case it was able to stay under 1 second.
    Not terribly bad, considering how much stuff happens in the background (TLS handshake, authentication and DNSSEC).

    Of course, after the initial request, the following DNS requests will give 0 ms because cached.... but only for the TTL that domain owner has specified.... unless you override that TTL ...

    There is cache-min-ttl (default value 0 seconds) mentioned in unbound.conf manual...
    https://www.unbound.net/documentation/unbound.conf.html

    EDIT:
    I set cache-min-ttl for 3600 (1 hour) for both client and server. Interesting to see what happens ...
     
    Last edited: Jul 24, 2018
  21. XIII

    XIII Registered Member

    Joined:
    Jan 12, 2009
    Posts:
    1,383
    Thank you for mentioning that the entries need to be moved.

    Had the same error, so will try this as well.
     
  22. XIII

    XIII Registered Member

    Joined:
    Jan 12, 2009
    Posts:
    1,383
    Moving the lines around my configuration is accepted and unbound does start.

    However, if I purposely make a typo in "dns.quad9.net" to test this, names still get resolved, so the protection does not seem to work?
     
  23. summerheat

    summerheat Registered Member

    Joined:
    May 16, 2015
    Posts:
    2,199
    Yes, my query times is about in the same range.

    Interesting! I'll try that, too.

    Well, I've been using several nameservers for quite some time from this site. However, I don't know if unbound does some load balancing like dnscrypt-proxy. Or does it send the request to all used nameservers and takes the fastest response?
     
    Last edited: Jul 25, 2018
  24. Stefan Froberg

    Stefan Froberg Registered Member

    Joined:
    Jul 30, 2014
    Posts:
    747
    Good catch I. Yea, it does same with me when giving bogust domain name and I think I found the reason.
    https://www.nlnetlabs.nl/bugs-script/show_bug.cgi?id=658#c16
    According to above link, the unbound need to be compiled against openssl 1.1.x for TLS authentication to work.

    In 1.7.3 there is no mention or warning or anything about that but at least in the github rebo the author has now added warning to logging code if authentication not working.
    https://github.com/NLnetLabs/unbound/blob/master/daemon/remote.c

    So, installing openssl 1.1.0h should fix this
    https://www.openssl.org/source/
     
  25. Stefan Froberg

    Stefan Froberg Registered Member

    Joined:
    Jul 30, 2014
    Posts:
    747
    Im not sure but according to this (https://www.monperrus.net/martin/randomization-encryption-dns-requests) it picks fastest server randomly within certain time:
    "Randomization is builtin in unbound: "The fastest server (randomly picked within a so-called RTT band of 400 msec) is selected when a query has to be sent out" source"

    Also, looking from this https://dnsprivacy.org/wiki/display/DP/DNS Privacy Clients#DNSPrivacyClients-Unbound tells that unbound does not yet re-use TLS connections
    (or other speed improvements like pipelining, out-of-order requests etc..) and some privacy features missing too.

    So I think latencies can still be improved in the future if new versions get those :)

    There is also a RFC draft of recommended speed improvements for DNS-over-TLS
    https://tools.ietf.org/id/draft-ietf-dprive-dns-over-tls-05.html
     
  1. This site uses cookies to help personalise content, tailor your experience and to keep you logged in if you register.
    By continuing to use this site, you are consenting to our use of cookies.