HTTP Switchboard for Chrome/Chromium:

I am thinking about running HTTP Switchboard together with an AdBlocker like AdBlock or ABP, while having HTTPSB's own blacklists deactivated. Any thoughts on that?

 

I just added this note in the wiki:

Note: January 24, 2014: The result for ADP+ must be dismissed. I found out there is something unorthodox going on with ADB+ which causes the benchmark to report erroneous results. I am investigating to figure exactly why a page error is fired for each page loaded while ADP+ is active, possibly explaining the low bandwidth and the overall better stats than a full-fledged blocker as HTTPSB OOB.​

I will investigate further when I have time. I did noticed subjectively that when the benchmark runs for ADB+, the CPU is working harder than with other blockers.

EDIT: I redid the benchmarks to fix erroneous ADB+ results. Now ADB+ bandwidth is no longer surprising, more inline with other blockers: https://github.com/gorhill/httpswit...st-Popular-News-Websites#wiki-january-24-2014

 

I didn't check the code yet, but it seems ADB+ modifies the header (given the number of outbound cookies reported), so this is where both might conflict as only one extension is allowed to modifies headers. I need to investigate if it is really the case and update the doc to warn users (so far I thought the conflict was just frame replacement, i.e. benign). If there is headers conflict, a user need to be made aware of which fonctionalities he would loose in HTTPSB depending on whether HTTPSB was installed before or after ADB+.

 

I am actually thinking if there are any conflicts with both extensions installed. Waiting for your findings.

 

I believe ABP on *Firefox* (bear with me for a second) has some behaviors worth noting, such as:

1) It ignores top-level requests and any redirects of those. So even if you have a ||nastydomain.example^ rule to block a domain, you can end up fetching something from nastydomain.example by clicking on a link to it or having a top-level request redirected to or off there.

2) Lists you subscribe to can contain exceptions that supersede not only blocking filters in the same subscription, but blocking filters in other subscriptions and even custom filters you created yourself. So although you may create and want to use ||nastydomain.example^, a subscription list could have an @@||nastydomain.example/advertisements/ exception that allows advertisements (or whatever) through because that was deemed necessary to prevent breaking something that average users wouldn't want to do without. This is also how the "allow some non-intrusive advertising" feature works I believe.

Such behaviors are not desirable when your objective is to definitively and fully block all requests to a blacklisted host/domain.

I can't test it, but I'm assuming that both of these "by design" behaviors are present in ABP for Chrome as well. Where such limitations exists, some would want to supplement ABP with another extension that provides more thoroughly blocking of blacklisted hosts/domains. It sounds like HTTPSB is more thorough, but again I am not testing anything Chrome/Chromium based.

 

Well this is it for now:
https://github.com/gorhill/httpswit...sed-blockers:-Most-recent-results-on-one-page

I will repeat the benchmarks in a couple of weeks to see what changed. What I can do for HTTPSB meanwhile is to work a better preset lists of blocked hosts which would benefit most the allow-all/block-exceptionally mode.

Observation from results above: Ghostery does better than Adblock+.

 

Weird stuff happening. Google on Wilders I refresh a couple times and Google isn't there. Close the browser, open it again to wilders and Google is back it seems on again, off again.

 

I'm glad that you bringing this up because I noticed something similar. No, I don't see Google on Wilders but I frequently open this site:

http://www.finanzen.net/

and use these rules in HTTPSB to make it work properly (site-specific scope):

Code:

http%3A%2F%2Fwww.finanzen.net%0A%09white
list%0A%09%09cookie%20*%0A%09%09image%20
*%0A%09%09object%20*%0A%09%09script%20pu
sh.finanzen.net%0A%09%09script%20scripts
.finanzen.net%0A%09%09script%20www.finan
zen.net%0A%09%09stylesheet%20*%0A%09%09s
ub_frame%20push.finanzen.net%0A%09%09sub
_frame%20scripts.finanzen.net%0A%09%09xm
lhttprequest%20push.finanzen.net%0A%09bl
acklist%0A%09%09*%20ioam.de%0A%09%09*%20
nuggad.net%0A%09%09sub_frame%20*%0A%09%0
9*%20*%0A

Whenever I load that site and subsequently open another one (even in a new tab!) like, e.g., www.nytimes.com, I'm seeing finanzen.net and push.finanzen.net in the matrix of, in this case, www.nytimes.com. How come?

At first, I thought it was a bug in HTTPSB. However, I also saw finanzen.net entries in the NYT list of blockable items in Adblock (without Plus; I prefer it as Adblock Plus for Chrome doesn't have that list in contrast to its Firefox version). In order to verify this observation I disabled HTTPSB, restarted Chrome, loaded finanzen.net and subsequently NYT - voilà: Those finanzen.net entries still existed in Adblock. After reloading the NYT site, those entries were gone. Thus, I think it's not related to HTTPSB but rather a Chrome bug.

 

Weird. This needs investigation.

An important note is needed. Though this doesn't explain why google.com would be present in the first place, it explains why it won't go away after reloading the page. HTTPSB cache the list of hit hostnames for a particular page up to 10 min after the page has been navigated away. I did that for performance purpose (existing dict of hostnames can be just reused instead of rebuilt), and so that the user can still look at whatever a page tried to load after switching to more restrictve rules for the same page.

Now re. google.com, looking at the request log should first confirm that a hit to google did really happen. And if you want to investigate further if it did really happen for the page itself by filtering the log for the wildersecurities page.

Now if it did really happen as per log, further investigation is needed of why it happened. Could there be an extension which inject code in the page itself? There is also the case of something a web page request being reported as being tabless (behind-the-scene), so I wonder if the reverse can also happen.

 

Thanks Raymond, I'll check later when I'm home. Maybe tlu is right that it might be a Chrome bug.

 

FWIW, I built this into my experimental toy. Firefox provides a DNS API via which you can retrieve the (final) canonical name of a host, so I included that information on each line. Some very limited testing with protection mechanisms backed off just a little bit confirmed that it was informative. I did spot a few more cases of a well-known site mapping one of their subdomains to a well-known ad/tracking subdomain. The later were .2o7.net or .omtrdc.net. I didn't investigate how well the CNAME was being cached by Firefox, but particularly in an environment where it is locally cached, I think one could run the CNAMEs past the blacklist too (when they are different than requestHost).

I didn't spot a browser interface to get at "intermediate" CNAMEs though. So in cases like:

scripts.example.com CNAME foo.adtracker.com
foo.adtracker.com CNAME blah.akamai.net

The foo.adtracker.com wouldn't be caught. It is one thing to frequently spot requests going to content delivery networks and cloud platforms. It is another to surf for awhile and then dump the list of hosts you contacted along with their CNAMEs. After this light testing by surfing to a few dozen well known sites on various subjects, it looked like roughly 85% or more of the requests were to Akamai machines. Pretty staggering, and scary frankly. Considering the possibilities for site1<->site2 sharing through APIs of the cloud they are both running on. Which simplifies the task and reduces if not eliminates the [external] traffic ($) necessary to carryout such information exchanges.

 

I must admit that I'm getting a bit confused of the behaviour of HTTPSB.

I've explicitly blacklisted some domains using a global scope:

Code:

*%0A%09whitelist%0A%09%09*%20lastpass.co
m%0A%09%09image%20*%0A%09%09object%20*%0
A%09%09stylesheet%20*%0A%09blacklist%0A%
09%09*%20ioam.de%0A%09%09*%20nuggad.net%
0A%09%09*%20plista.com%0A%09%09*%20vgwor
t.de%0A%09%09*%20*%0A

Now let's see what happens once I load several websites like http://www.faz.net/. My site-specific rules for this site are:

Code:

http%3A%2F%2Fwww.faz.net%0A%09whitelist%
0A%09%09cookie%20*%0A%09%09image%20*%0A%
09%09script%20*%0A%09%09stylesheet%20*%0
A%09%09xmlhttprequest%20www.faz.net%0A%0
9blacklist%0A%09%09*%20*%0A

Nevertheless, the matrix for this site shows both for ioam.de and plista.com light-green cells in the cookie, css, image and even script columns! Note that the narrower site-specific scope does not whitelist them.

Another site is http://www.spiegel.de/ . My site-specific rules are:

Code:

http%3A%2F%2Fwww.spiegel.de%0A%09whiteli
st%0A%09%09cookie%20spiegel.de%0A%09%09i
mage%20spiegel.de%0A%09%09script%20www.s
piegel.de%0A%09%09stylesheet%20spiegel.d
e%0A%09blacklist%0A%09%09*%20*%0A

Here all cells for ioam.de are light-red. This is okay - but shouldn't the cell in the site column rather be dark-red since I blacklisted that domain using a global scope?

In any case the behaviour in both cases is inconsistent, IMHO. And in the first example I wouldn't expect any green cells for those blacklisted domains at all.

Raymond, can you reproduce what's happening here?

 

There is no inheritance across scopes. So whatever is blacklisted in the global scope won't be inherited by a narrower scope. Currently what is happening as mitigation is that the blacklisted hosts will be copied at scope creation time (not import time though). But if you blacklisted after, this won't propagate to your narrower scopes.

Inheritance across scopes has been raised before, and each time I entertain the idea, what I call intractable complications arise. Currently a single cell in the matrix inherit from two paths: its hostname on the left and its type on top. The hostname itself might inherit from the domain name if it is not itself a domain meaning this will also apply to the cell. Etc. Eventually, all cells ultimately inherit from the master "all" cell, if there is no rule in between.

Now to add inheritance from a higher-level scope I am just unable to envision how that would work: in the inheritance order, when would a cell lookup the larger scope rather than the master cell or the domain cell or the type cell? This is impossible to solve.

Hence the proper way to look at it is that all scopes are sandboxed, they do not know about each others.

Now I agree there is a confusion and it is rooted in the fact that I show the preset-blocked hosts the same way I show a user-blocked host. Since the preset are omnipresent, the user expects the same thing to happen to his blocked host.

Now there is this issue I entered 10 days ago: https://github.com/gorhill/httpswitchboard/issues/152, which is to allow the user to create omnipresent rules, i.e. rules which apply in every scope. I didn't settle yet as to how I will implement this. This would solve your problem.

 

To just continue my thoughts on this.

Well I could put any user-blocked host in the same collection as the preset blocked hosts, and this way your own blocked hosts would be seen by all scopes.

If I could figure a visual and interaction in the matrix for "omnipresent" rules to be visually distinct and interactively set (would only apply to hostname though), that would be a great improvement and this would address the confusion. I must admit I lack the inspiration as to how to represent and handle the interactivity for omnipresent rules in the matrix.

 

For version 0.7.8.0, there will be a new feature, the preset recipes accessible from the popup matrix. I already wrote down recipes for embedded Youtube, Vimeo and Disqus. But it's a bit too few I fear. I would like to ship this feature with a bit more of those recipes, the ones likely to be convenient to as many people as possible. So if anybody has ideas for which recipes (likely to be useful to as much people as possible) should be shipped in v 0.7.8.0, I would appreciate the suggestions.

(I use "recipe" but the internal syntax is not the same as the ones in the Rule manager. The syntax is more human friendly, you can see here).

 

@gorhill: Do you know if there is any leakage possible when using HTTP Switchboard with HTTPS Everywhere?

 

You mean "http:" requests in the context of a "https:" container?

I didn't test HTTP Everywhere, but I will activate it. My understanding is that Chromium catches and blocks requests to an unencrypted connection when on a "https:" page .

 

Yes. I don't use HTTP Switchboard because I use Firefox, but I thought some might want to know if the rewrite rules in HTTPS Everywhere can result in leakages.

 

Thanks, I understand the problem. I hope that you will find a solution for omnipresent rules.

 

Great feature V. 0.7.8.0 is already out, but for the next version I would suggest to add a recipe for brightcove.com which is used on many sites (particularly newspapers) in order to display videos.

EDIT: Plugin must be allowed for admin.brightcove.com and c.brightcove.com, scripts must be allowed for admin.brightcove.com, api.brightcove.com and sadmin.brightcove.com.

 

Note: These are two different "jobs"...

1) Developing the tool, adding the features necessary to allow users to accomplish their goals, keeping up with Chrome/Chromium changes and working with those developers if/when an API limits your features, etc.
2) Maintaining rules and lists for a climate that is forever changing, for users who have substantially different tolerances and sites they do/don't care about, etc.

The first is a product, the later is more of a service. Suggestion: Keep #2 as separate from the extension and distributable as you can. A mechanism which makes it easy to pull "recipes", "blacklist updates", etc... from anywhere or anyone and without having to update the extension will be far more valuable to users in the long run, and easier for you too. Unless you want your project to be a service.

 

I completely agree with this, and I am still trying to figure how to get as close as possible from relieving myself from the work of updating what I call the "assets". Updating all the assets now is taking more and more time, which I rather spend on coding.

 

Do you have one or more site where I can test embedded brightcove.com?

 

Will this help:

http://www.calgarysun.com/videos/3102659276001

 

Another ones are http://www.nytimes.com/ and http://www.handelsblatt.com/