Chromium: network requests bypass blockers

Discussion in 'other software & services' started by summerheat, Sep 28, 2019.

  1. summerheat

    summerheat Registered Member

    Joined:
    May 16, 2015
    Posts:
    1,698
    I'm using Pi-hole (running on a Raspberry Pi) as my DNS server. It uses several hosts files (and you can add even more) which block network requests to countless ad networks, trackers and malware sites system-wide. Its query log displays all network request and shows which of them are blocked or not. (Btw: an alternative solution is AdGuardHome).

    That query log is the reason why I can confidently say that uMatrix and uBlock Origin block/filter network requests reliably on Firefox which has been my default browser for a long time. But what about other browsers?

    Let's perform a test with Chromium and uMatrix and uBlock Origin (or with uBO alone in Hard Mode blocking mode if you want) and load www.washingtonpost.com. This is what the Pi-hole query log shows us:
    upload_2019-9-28_18-12-43.png

    Oops! These are requests that should have been blocked by uM and/or uBO (and are now blocked by one of the hosts files in Pi-hole). However, if you started the uM or uBO logger before loading that site, the requests to, e.g., outbrain.com do not show up at all.

    Is this a problem of uM or uBO? I created a new Chromium profile and installed AdBlock Plus instead (and added some additional lists like EasyPrivacy). But the result is the same as above: There a blocked requests in Pi-hole which should have been blocked by the add-on.

    I repeated the tests with the Chromium-based browsers Vivaldi and Brave: The situation is no different than before which is especially interesting as both browsers come with their own built-in adblockers which were not able to filter these requests, either.

    Again, this problem does not occur in Firefox - but why in Chromium (-based browsers)? The answer is given in the uBlock Origin wiki: uBO offers the option to "Disable pre-fetching" (which was checked in my tests, of course). The wiki says:
    And now the important part:
    @gorhill found out that
    Note that a duplicate of the related Chromium bug is closed as "fixed". So either this fix is incomplete - or there is another bug that allows for this bypass.

    Note that I observed these bypasses only on few domains, and www.washingtonpost.com is one of them. But many other sites could also be affected - and most users don't notice because they do not use something like Pi-hole. This solution or a big hosts file in your system could mitigate these bypasses.

    Conclusion: If you care for your privacy don't use Chromium (-based browsers). Manifest 3 will make the situation even worse.
     
  2. Rasheed187

    Rasheed187 Registered Member

    Joined:
    Jul 10, 2004
    Posts:
    12,041
    Location:
    The Netherlands
    So you're saying that Google has already made certain changes to Chromium? I wonder if these changes can be somehow stopped by Brave and Vivaldi.
     
  3. summerheat

    summerheat Registered Member

    Joined:
    May 16, 2015
    Posts:
    1,698
    No, this rather seems to be a bug which hasn't been fixed in years.
     
  4. Yuki2718

    Yuki2718 Registered Member

    Joined:
    Aug 15, 2014
    Posts:
    1,757
    Thanks for head up, Can you confirm if NetworkPredictionOptions policy doesn't change the behavior? All uBO users should be aware that its blocking capability is severely neutered on Chromium. You know, if it was Firefox we had a last resort - removing the ad-calling element from the document by HTML filtering. But we need to inject a script just to block an inline-script on Chromium!

    I've been searching for effectiveness of blockers (a part of results are in this thread), and am almost desperated to see too many bypass techniques and the fact popular blacklists only covers a small fraction of trackers - does it even reach the half? Even an amateur like me can easily spot some trackers slipped through medium mode + various lists.

    Anyway, for this particular issue the easiest workaround would be using ad-blocking DNS. I personally don't like this and network-level blocking, as they make FP hunting harder - given I come across FPs almost every week (about 60% of them are by EasyPrivacy), it's convenient to limit the cause to one place (browser) - not to mention domain-level blocking can't block trackers like yimg.com/ss/rapid*, but I'll consider some measures as I use Brave (as well as Firefox).

    This paper is good to glance at bypass techniques: https://gangw.cs.illinois.edu/imc2019-adblock.pdf
    This is the most comprehensive filter comparison: https://web.wpi.edu/Pubs/E-project/Available/E-project-032216-001707/unrestricted/dcuzunoglu_understanding_ad_blockers.pdf
    Another good read about AAB and lists: https://conferences.sigcomm.org/imc/2017/papers/imc17-final113.pdf
     
  5. Yuki2718

    Yuki2718 Registered Member

    Joined:
    Aug 15, 2014
    Posts:
    1,757
    Are you aware that uBlock filters have separate code chunks for Chromium? This is because uBO has been neutered on Chromium. gorhill has repeatedly warned limitations of uBO on Chromium. Webmasters can easily bypass ad-blockers by moving ads/trackers to their own site, and indeed I've seen some 1st-party ads & trackers. You can remove them on Firefox, while on Chromium you can block only a part and just hide the rest (ads are loaded but invisible).
     
  6. SouthPark

    SouthPark Registered Member

    Joined:
    Jun 13, 2012
    Posts:
    384
    Location:
    USA
  7. Brummelchen

    Brummelchen Registered Member

    Joined:
    Jan 3, 2009
    Posts:
    2,954
    beside that WP is pain for any blocker - thats reason why i use a HOSTS file for all of my systems (similar to pi-hole filter) as a basic protection.

    and no - although google is involved into chromium code its not "their" product - google offers "Chrome" which includes closed source and more modifications. chromium is open source.

    nevertheless @gorhill still had clear words about uBo limitations.

    btw uB Extra
    https://github.com/gorhill/uBO-Extra/wiki/Sites-on-which-uBO-Extra-is-useful
     
  8. Yuki2718

    Yuki2718 Registered Member

    Joined:
    Aug 15, 2014
    Posts:
    1,757
    Note uBO-Extra is for Websocket/WebRTC bypasses and a nasty script technique to disguise 3rd-party requests as 1st-party, thus has nothing to do w/ what summerheat & I are talking.
     
  9. summerheat

    summerheat Registered Member

    Joined:
    May 16, 2015
    Posts:
    1,698
    I added
    Code:
    {
      "NetworkPredictionOptions": 2
    }
    
    to the policy but it didn't make a difference. :thumbd:

    Indeed. Inline script tag filtering does not work in Chromium-based browsers.

    I'm trying to answer in that other thread. Very briefly: I'm not that desperate as by combining multiple lists most trackers are covered. But definitely the best thing one can do is using uMatrix or uBlock Origin with Dynamic Filtering in medium mode or, better yet, hard mode - but not everyone is willing to do this as it requires more micro-management.
     
  10. wat0114

    wat0114 Registered Member

    Joined:
    Aug 5, 2012
    Posts:
    2,557
    Location:
    Canada
    Thank you @summerheat for this discovery and sharing it. Also good to see you back ;)

    Well I guess I may have to bite the bullet and use Firefox instead, because this is something that even I can't really tolerate :mad: The only thing that annoys me with Firefox is it will not play Netflix content in Windows 10 for me, no matter what solutions dug up from around the 'net I've tried. Stupid Widevine crashes every time without fail. It does, however, work for me from my Linux MX-18 pendrive install no problems.

    Thanks again!
     
  11. summerheat

    summerheat Registered Member

    Joined:
    May 16, 2015
    Posts:
    1,698
    You're welcome!

    I can't say anything about the Netflix problem as I don't use it. Perhaps someone else? (Preferably in its own thread)
     
  12. The Red Moon

    The Red Moon Registered Member

    Joined:
    May 17, 2012
    Posts:
    4,036
    Talk about shooting yourself in the foot google.This will lose them a ton of users.
     
  13. wat0114

    wat0114 Registered Member

    Joined:
    Aug 5, 2012
    Posts:
    2,557
    Location:
    Canada
    Absolutely another thread if I pursue this again.
     
  14. gorhill

    gorhill Developer

    Joined:
    Nov 12, 2013
    Posts:
    837
    Location:
    Canada
    Chromium is what Google decides it is and will be.
     
  15. summerheat

    summerheat Registered Member

    Joined:
    May 16, 2015
    Posts:
    1,698
    Definitely!

    Is what I wrote in post #1 correct to your knowledge or are there new insights why these bypasses occur?
     
  16. wat0114

    wat0114 Registered Member

    Joined:
    Aug 5, 2012
    Posts:
    2,557
    Location:
    Canada
    Agreed. This is like a kick in the gut. I've long stuck with Chromium-based browsers because I like its sandboxing security architecture. This latest revelation is the straw that broke the camel's back for me.
     
  17. Yuki2718

    Yuki2718 Registered Member

    Joined:
    Aug 15, 2014
    Posts:
    1,757
    Thx for confirmation, so I seriously need to seek for mitigation. I hope Brave one day plugs the hole but that won't come soon.
     
  18. Yuki2718

    Yuki2718 Registered Member

    Joined:
    Aug 15, 2014
    Posts:
    1,757
    BTW, even after Manifest v3 those who rely 100% on subscription filters won't be much affected - at least ABP will survive (not sure how element hiding will become). Those who'll be affected are ones using advanced capability of uBO, uMatrix, ScriptSafe etc. At first I misunderstood the problem as a matter of # or rules, but now I see the real problem is that the new API must be declarative. I hope the blocking capability of WebRequest API will be still available via local policy, but maybe nobody knows - does the manifest has already applied to Canary?
     
  19. Rasheed187

    Rasheed187 Registered Member

    Joined:
    Jul 10, 2004
    Posts:
    12,041
    Location:
    The Netherlands
    So far I haven't had any big problems with ad blocking done by uBlock. However, what does bug me is that more and more sites are able to spot that you're using an ad-blocker. I recently tried ABP and it was often detected but I think I forgot to enable the ABP Anti-Circumvention Filter List.

     
  20. Yuki2718

    Yuki2718 Registered Member

    Joined:
    Aug 15, 2014
    Posts:
    1,757
    That's depends on what is a big problem to you. If you're happy w/ just not seeing ads, that's okay. But remember ads are loaded, so you're tracked, performance is degraded, bandwidth is consumed, and you may be hit by malvertising. IDK what you mean by detection, but as a paper I posted in another thread reported, many websites detect ad-blocker (and yes, ABP is much more detected than uBO) and switch their content. But you'll never notice them unless you compare the sites w/ & w/out ad-blocker and inspect them by e.g. dev tool. So I guess you mean ad-blocker warnings and AAB. In my case I don't care warnings and rarely come across AAB. Even when came across, it usually doesn't take a minute to remove them.

    I haven't examined ABP-AC filter as I don't use ABP, but it will have nothing to do w/ detection. If you don't like warnings, subscribe Adblock Warning Removal List. If you come across AAB frequently but don't know how to address it, Nano Adblocker may be one option. Of note, uBO's uBlock filters block circumventing ads and AAB too.
     
    Last edited: Oct 6, 2019
  21. summerheat

    summerheat Registered Member

    Joined:
    May 16, 2015
    Posts:
    1,698
    I don't know if Nano is really better in that respect. In any case it'a also possible to integrate this with uBO.

    On the hand I'm a bit cautious about a develper who maintains a widely used fork of uBO but writes:
    That doesn't really count as a letter of recommendation ...
     
  22. Rasheed187

    Rasheed187 Registered Member

    Joined:
    Jul 10, 2004
    Posts:
    12,041
    Location:
    The Netherlands
    Yes, this is a problem of course. BTW, here are some sites that can spot that I'm using uBlock and/or ABP:

    https://www.filehorse.com/
    https://www.funnygames.nl/spel/deal_or_no_deal_1.html
     
  23. Yuki2718

    Yuki2718 Registered Member

    Joined:
    Aug 15, 2014
    Posts:
    1,757
Loading...
  1. This site uses cookies to help personalise content, tailor your experience and to keep you logged in if you register.
    By continuing to use this site, you are consenting to our use of cookies.