µMatrix - the HTTP Switchboard successor

Discussion in 'other software & services' started by tlu, Oct 25, 2014.

  1. gorhill

    gorhill Guest

    An example of the benefits can be seen with uMatrix's default settings. By default all CSS and image resources are allowed. However, CSS and image resources from blocked hostnames won't be allowed, so the benefit is that should you allow a whole class of resources, or even the `all` top-left cell, you will still be well protected against known bad hostnames (ads, trackers, etc.). It just give you more flexibility about what you can safely allow.

    Another way to see the benefit, is that when working in allow-all/block-exceptionally mode, the blocked hostnames is what protects you. Even if you work in block-all/allow-exceptionally mode, with a single click you may want to go allow-all/block-exceptionally mode for a specific web site (let's say you give up on trying to un-break it) by toggling the `all` cell to allow (green) for the current site. The blocked hostnames will still offer you a good protection -- not everything will really be allowed.
     
  2. glopglop

    glopglop Registered Member

    Joined:
    Aug 29, 2015
    Posts:
    8
    Thanks gorhill for this complete explanation.
     
  3. summerheat

    summerheat Registered Member

    Joined:
    May 16, 2015
    Posts:
    2,199
    @gorhill: I wonder if you're thinking about re-integrating pattern-based filtering again.

    I've read a lot of posts on your HTTP Switchboard Github site. As far as I understand there were 2 reasons why you removed pattern-based filtering in uMatrix:
    1. Pattern-based filtering didn't really fit in with the matrix-based filtering.
    2. You wanted to have a smaller code for easier maintenance.

    I absolutely understand these reasons. However - hasn't the situation changed a lot since then?

    1. uBlock Origin is no longer the "simple" blocker for the masses but has been enhanced considerably. Dynamic filtering is something of a "uMatrix light". And you've demonstrated how to beautifully integrate pattern-based and matrix-based filtering.
    2. With Dynamic URL Filtering you've even introduced a feature which can result in odd situations if you're using uBlock0 and uMatrix together. On https://github.com/gorhill/uBlock/wiki/Dynamic-URL-filtering you wrote : "The primary purpose of dynamic URL filtering ("URL filtering") is as a diagnosis/mitigation tool to fix web page breakage caused by dynamic filtering or static filtering." Now take your example from the bottom of that page: Say, I want to completely block https://foo.com but have to allow https://foo.com/widget.js in order to unbreak the current website. This means, I have to whitelist "script" for foo.com in uMatrix, block scripts for foo.com in Dynamic Filtering in uBlock0 and allow https://foo.com/widget.js in Dynamic URL Filtering. Needless to say that this would be much more straightforward with an integrated extension.
    3. From your comments and commits in the past months I have the impression that uBlock0 and uMatrix share a relatively large part of their code (particularly what is vAPI and CSP related). I don't know how large that part is but I assume that maintaining the code is no longer that painstaking.

    The advantages of a uMatrix with integrated pattern-based filtering are obvious, IMHO. There would be one extension with one logger which would make it much easier to find the "culprit" if a website doesn't behave as expected. And there would be a clear distinction between an extension (uBlock0) that is aimed at a large number of users and an extension (uMatrix) that is perfect for advanced users who wish very granular control and offers everything what uBlock0 has.

    My idea would be that the "ternary" approach (as you called it somewhere) would also be available in the first column in uMatrix for each (sub)domain. It should still be possible to independently whitelist/blacklist every cell.

    I think that would be quite simply the ultimate blocker!

    Perhaps you think that what I've written above is nonsense or that I underestimate the problems and the work involved. I accept that, of course. It's your free time you're spending, and we are more than grateful for what you're doing for us. Nobody has the right to demand anything from you. It's just some thoughts I was trying to present, and I would be interested in your opinion.
     
  4. AutoCascade

    AutoCascade Registered Member

    Joined:
    Feb 16, 2014
    Posts:
    741
    Location:
    United States
    Gorhill in an old thread you mentioned that Chrome only allowed one extension to modify headers and that the last extension added will be the one to have that function. I'm curious if something like HTTPS Everywhere is in conflict with uMatrix?
     
  5. mkewU

    mkewU Registered Member

    Joined:
    Apr 4, 2015
    Posts:
    18
    Anybody know if uMatrix offers any protections similar to NoScript's ABE system, specifically the rule that guards the local network by matching all LAN subnets:
    Code:
    Site LOCAL
    
    Accept from LOCAL
    
    Deny
    This rule always provided me some measure of comfort and I'm wondering if there's a similiar configuration I can work out using uMatrix.
     
  6. summerheat

    summerheat Registered Member

    Joined:
    May 16, 2015
    Posts:
    2,199
    I've never noticed any conflicts between both extensions. AFAIK, HTTPS Everywhere doesn't modify headers.
     
  7. anniew

    anniew Registered Member

    Joined:
    Mar 15, 2013
    Posts:
    92
    Just started testing uMatrix and uBlock Origin recently, as I really like the flexibility of the greater granularity/visibility of controlling what is happening on a given website.

    However, I ran into an issue with a particular website.

    It seems excessively difficult to nail down which are the elements I need to allow a specific (desired) function to run via some select scripts.

    On default uMatrix and uBlockO (I've not added lists, tweaks), the third party script necessary for the functionality just doesn't show up in uMatrix to authorize.

    It only shows up after I allow BOTH the 1st party XHRs (at this point 7 of them - the log filtered on XHR shows eight, two sets of four) AND have turned off uBlockO (blocking 8 requests). Incidentally, the log in uBlockO shows those same four 1st party XHRs, and allows them.

    Turning off uBlockO alone doesn't do much, probably because those 8 now unblocked requests are all from domains blocked in uMatrix.

    Likewise, neither turning those 1st party XHRs alone, nor turning off uMatrix completely does much, while uBlockO is on.

    If I turn off uBlockO, AND turn on the XHRs in uMatrix, I get the domain and it's script available in uMatrix to authorize. The result is a lot of extra garbage littering the website.

    If I turn uBlockO back on, and turn off those XHRs, that domain and script remain listed (as do many, many others in uMatrix, thankfully, most default blocked, while uBlockO is now preventing 60+ requests) and I am able to access the functionality. Can only detect one new unwanted element on the web page that was not there with the original settings, btw.

    Of course doing this defeats the purpose of blocking in the first place.

    It looks like the only path to the granularity I'm looking for is to individually write rules, in a trial and error between uMatrix and uBlockO to attempt to find the culprits.

    Instead, would rather have the uMatrix one click on/off capability for the individual elements in the logs for each uMatrix and uBlockO. Otherwise, too much work for find the right combo (after already spending a lot of time with trial and error at the cell level blocking).

    Am I missing something obvious? (Probably!)
     
  8. wat0114

    wat0114 Registered Member

    Joined:
    Aug 5, 2012
    Posts:
    4,063
    Location:
    Canada
    Running two blockers concurrently, no wonder why you are running into broken website problems.
     
  9. summerheat

    summerheat Registered Member

    Joined:
    May 16, 2015
    Posts:
    2,199
    You didn't tell us how exactly you're using uBlock0. I assume that you're using Dynamic Filtering in uBlock0. In this case it's not surprising that both extensions get in each other's way. You should disable Dynamic Filtering in uBlock0 and restrict it to static filtering (aka filterlists).
     
  10. Jarmo P

    Jarmo P Registered Member

    Joined:
    Aug 27, 2005
    Posts:
    1,207
    +1
    When operating uMatrix, the notion that you are an advanced user should maybe not be ticked in uBlock Origin. I am not in knowledge of how to to make special filters in uBlock in some special site, so I leave that out, but in general: Leave dynamic filtering to uMatrix's control. It will be a total nightmare to have also some default deny in uBlock Origin same time.

    Some sites have hidden stuff with Strict HTTPS option if selected in uMatrix, so that is of course an option to disable it site specific, when wanting to see what that stuff is. I noticed that even if you disable matrix filtering for a scope, that http stuff will still be filtered in that scope. There is the 3 dot icon popup to disable it site specific. That is just another thing that came to my mind.
     
    Last edited: Sep 3, 2015
  11. anniew

    anniew Registered Member

    Joined:
    Mar 15, 2013
    Posts:
    92
    Thanks for your replies.
    I've read a few places that these two tools have different aims and are complementary. uBlockO provides Dynamic Filtering which uMatrix does not.
    https://www.wilderssecurity.com/threads/ublock-vs-umatrix.376016/
    I am using the defaults on both uBlock0 and uMatrix.

    Actually, rereading the link above, I forgot the part about the filterlists... however, in that thread it gives the opposite advice...
    In any case, it doesn't matter, as when I turn uBlockO completely off, I still encounter the problem.
    Advanced user is not checked to on in uBlockO. Only default is used on both tools.

    My understanding is that uBlockO does the dynamic filtering, not uMatrix.
    Besides, as mentioned, when I turn off uBlockO completely and only run uMatrix, I still have the problem.
    Good point, and something to look for later, but with the default setting in uMatrix, Strict HTTPS is not on.

    My guess from the responses is that users (from this community) have their setups highly customized, and have long since reached beyond the defaults.

    I may have complicated matters by mentioning uBlockO, as, primarily, it comes back to the XHRs in uMatrix, if I did not have uBlockO.

    The problem without uBlockO, once the 1st party XHRs are opened, is the volume of "garbage" on the screen page.

    It is a catch 22. In either case (with or without uBlockO) I have to turn off a protection, to get to what I want. Strangely, that domain and script are accessible even when I turn that protection back on (uMatrix "remembers" them for the Firefox session, but "forgets" them when Firefox is closed and later reopened. And, uBlockO never has a problem with them specifically - if uMatrix "knows" the domain).

    Hence, it seems that there is a need to be able to selectively authorize elements within a cell (or via the log) to see what is the culprit XHR in uMatrix. Or, maybe, it just needs to permanently remember the domains with requests and their authorization settings for a given 1st party domain.

    Likewise, if I wish to run uBlockO, it seems one of the blocked elements is its own culprit (since I can turn off uMatrix completely, but still not have the functionality available), thus it can use the ability to selectively authorize an element as well.

    The only capability available in either tool is to manually write rules - too much effort for a trial and error process of finding the right combination.
     
    Last edited: Sep 3, 2015
  12. summerheat

    summerheat Registered Member

    Joined:
    May 16, 2015
    Posts:
    2,199
    No, I don't think so. The suggestions made in that thread regarding dynamic filtering were a for uBlock0 used without uMatrix. Anyways - what's called "Dynamic Filtering" in uBlock0 is actually a sort of "uMatrix light". Or in other words: uMatrix is Dynamic Filtering on stereoids with a much more granular control.

    Regarding your specific problem, it's difficult to help you since you didn't provide a specific website and what you did exactly.

    EDIT: In order to see what is blocked you should open the logger in uMatrix.
     
  13. Jarmo P

    Jarmo P Registered Member

    Joined:
    Aug 27, 2005
    Posts:
    1,207
    Like summerheat told, you have got it a bit wrong way about what is dynamic filtering.

    Originally when HTTP Switchboard extension in Chrome was divided into 2 parts. uMatrix was to handle all the dynamic user dependent input though it still had the hosts files that are static filtering. And uBlock to be an easy adblocker with static, 3rd-party filters. The dynamic filtering got back into also uBlock by the demand of people wanting more control than the pure static filtering could provide.

    Think of default deny as where you need to allow things yourself as dynamic. I agree that you should give an example, site and what is blocked. Short and clear.

    "The problem without uBlockO, once the 1st party XHRs are opened, is the volume of "garbage" on the screen page."
    That I can understand. uBlock has some filters that hide blocked things. That is one reason I certainly use uBlock (without dynamic filtering enabled) with my uMatrix. The xhr thing without an explicit example though goes a bit over my head.
     
    Last edited: Sep 3, 2015
  14. anniew

    anniew Registered Member

    Joined:
    Mar 15, 2013
    Posts:
    92
    Fair enough, and thanks for the explanation.

    And, I combed through uBlockO and do see that Dynamic Filtering is available, but only if one clicks "Advanced User" option (by default it is not on).

    I was going by this statement, which I guess is confusing the point, as a read more into it than was warranted...
    So, as I have it set up, both tools are essentially filtering based on their default (mostly 3rd party) lists, of which there is overlap.

    Same here... no dynamic filtering enabled on uBlockO.

    Thanks. Have been very much in logger on both uMatrix and uBlockO.

    The XHRs in question are all js calls through 3rd party sites, mostly taboola.com (hence the garbage) - but that host name is not on the list of uMatrix domains. Seems one must turn the XHRs on to get access to the target domain, then turn them off again to clear up the page.

    Allowing the XHRs unleashes all kinds of other requests. There seems to be successive layers of elements called that eventually bring up the target domain / host site. Bottom line, for this site, anyway, uMatrix stand alone is not workable, as I'm not about to allow taboola through.

    I understand about knowing the website to have your own look at it. Unfortunately, would like to get specific regarding the site, but not with this audience. I suspect this is not a single anomaly, so if I find one appropriate will report it so all can have a go at it.

    Perhaps for another thread, but I tried NoScript and uBlockO on the same web page as used above, and ran into a problem where refreshing the web page directly from the browser tool does not change anything, but when refreshing via uBlockO's logger page (not the browser's own refresh), seems the target functionality (and other elements) are allowed through. Strange.
     
    Last edited: Sep 3, 2015
  15. wat0114

    wat0114 Registered Member

    Joined:
    Aug 5, 2012
    Posts:
    4,063
    Location:
    Canada
  16. dogbite

    dogbite Registered Member

    Joined:
    Dec 13, 2012
    Posts:
    1,290
    Location:
    EU
    I have a problem on Youtube. With these settings (see screenshots) I am not able to see the comments.
    Any idea?
     

    Attached Files:

  17. Jarmo P

    Jarmo P Registered Member

    Joined:
    Aug 27, 2005
    Posts:
    1,207
    Allow cookies from youtube.com. I know it sucks and have changed, but allowing them anyways works for me.
     
  18. dogbite

    dogbite Registered Member

    Joined:
    Dec 13, 2012
    Posts:
    1,290
    Location:
    EU
    Damn...I don't like it.
     
  19. Wroll

    Wroll Registered Member

    Joined:
    Nov 29, 2011
    Posts:
    549
    Location:
    Italy
    If you're on firefox you could use the Self-Destructing Cookies addon if you're worried about tracking.
     
  20. summerheat

    summerheat Registered Member

    Joined:
    May 16, 2015
    Posts:
    2,199
    Neither do I. However, if you allow cookies for Youtube in uMatrix but allow only session cookies in your browser it's acceptable, IMHO.
     
  21. summerheat

    summerheat Registered Member

    Joined:
    May 16, 2015
    Posts:
    2,199
    Well, gorhill himself is not happy with that article which was edited by someone else. I think it's rather easy: Do not enable those hosts files in uBlock0 which are also enabled in uMatrix (because if you whitelist a blacklisted 3rd-party domain in uMatrix in order to unbreak a site, it would still be blocked in uBlock0), and do not use Dynamic Filtering in uBlock0. That's it, IMHO.
     
  22. wat0114

    wat0114 Registered Member

    Joined:
    Aug 5, 2012
    Posts:
    4,063
    Location:
    Canada
    Raymond (gorhill) edited the page on June 21, although I'm not sure it now accurately reflects his views.
     
  23. dogbite

    dogbite Registered Member

    Joined:
    Dec 13, 2012
    Posts:
    1,290
    Location:
    EU
    Well...given the quality of comments on youtube maybe I stick without cookies.:D:D
     
  24. wat0114

    wat0114 Registered Member

    Joined:
    Aug 5, 2012
    Posts:
    4,063
    Location:
    Canada
  25. paulderdash

    paulderdash Registered Member

    Joined:
    Dec 27, 2013
    Posts:
    4,639
    Location:
    Under a bushel ...
    @gorhill - Could you please clarify? It is not clear if the page is now correct, or still needs to be edited.
     
  1. This site uses cookies to help personalise content, tailor your experience and to keep you logged in if you register.
    By continuing to use this site, you are consenting to our use of cookies.