Matousec Updates Firewall Challenge

Looks like Norton Internet Security 2009 improved just about everything on NIS
http://www.matousec.com/projects/firewall-challenge/results.php

 

Thanks for the link, but did you notice that matouse FWIW there posted

Best to stick with the recommended list I think.

 

Yes I did I was only pointing out the improvement.Previous version Very Poor 32% (against 62 tests), the new version comes with Good protection level and final score of 71% (against 73 tests). I used Comodo for almost 2 years and after chasing down so many pop ups that turn out to be nothing. I went with another of their top picks. Only put up with that for a month and decided that a firewall/HIPS wasn't for me. I've never had a virus that wasn't caught as it was trying to infect my computer. So no more over kill for me. In the past I've used Norton 2003, then AVG both with windows firewall. Then Avira free and premium with Comodo. Then Nod with Comodo. Whatever combo I used kept my computer clean. Not once did I need the extra protection of Comodo. So in my case Norton should be just the ticket no more needless pop ups.

 

Matousec's recommendations don't really matter. If a product can favorably meet the particular requirements of a given individual, then it is an excellent product for that individual. My router would fail miserably at his poc tests, but I would not give it up for any other software security license I own, including three of my favorites: Outpost, Jetico or System Safety Monitor. He rates the latter two products, not coincidentally, "not recommended". I don't care. IMO they are "excellent". For my needs they provide the security I'm looking for, and then some.

 

the way he test is so flawed that a it could not hold up to a fly.

 

A firewall test that doesn't test the capabilities of a firewall. YAY!
A firewall test that tests software that... isn't a firewall? WTF?

Matsousec was who I turned to WAY back in the day, but it's just a joke now that I know better. If you run a separate HIPS, pretty much every *firewall* on that list would be perfect. The fact that he tests HIPS programs that don't even include a firewall is just hilarious.

 

I'm not an advance user. I did read the tests he performed and to me it seemed like it was much more than just a firewall test. In the future if I fill I need more protection I could always add TF. Don't get me wrong Comodo was great. I don't think anything went on in my system without Comodo giving me a pop up.
If I did get infected I'm sure Comodo would have alerted me. The problem wasn't Comodo it was me looking at the alert and making a Homer Simpson comment to myself.

 

if it means anything according to the Matousec results the PCtools firewall went from 6% in version 3 to now 85% in version 4.

 

1 have 1 issue only with Matousec's test and that's the reccomendations. How is it that products rated as good are NOT RECOMMENDED - beats the .... out of me!

 

The ones he recommends are the ones he has a referral link to, so he makes money on every purchase. According to the affiliate plans from Outpost, up to 50% of the sale is commission.

 

Yup, exactly. Vendors who don't agree upon such pretty weird (for "independent" testing area) agreement or who offer free products (Comodo) do not get "recommendations" no matter what their result is. Matousec is about money and nothing else.

 

How is the test flawed?

Don't those HIPS have a outbound firewall?

Thanks

 

He's not testing firewalls in the first place. The whole "firewall challenge" is a misnomer.

 

Thanks for the explanation. Are leak tests considered something an outbound firewall without HIPS should block? Wouldn't the driver and OS crashes be related to a firewall?

 

Take a look at what some of the tests cover, they're all over the place. Why should a firewall have to detect a keylogger? let's take a look at his tests at level 7:

BITStest
Test type: Leak-test
Scoring: Failure – 0%; Success – 100%.
Description: BITStest checks if it is possible to bypass your firewall by using Background Intelligent Transfer Service (BITS) through Background Copy Manager COM interface.

Keylog5
Test type: Spying test
Scoring: Failure – 0%; Success – 100%.
Description: Keylog5 repeatedly attaches the keyboard input of the foreground windows' process to be able to sniff the user's input of that process.

Keylog6
Test type: Spying test
Scoring: Failure – 0%; Success – 100%.
Description: Keylog6 registers a raw input device to be able to monitor user's keystrokes.

OSfwbypass
Test type: Leak-test
Scoring: Failure – 0%; Success – 100%.
Description: OSfwbypass tests whether your firewall is able to deny an attempt to display and execute contents of a malicious HTML page. This attempt is performed by a special API.

Runner2
Test type: Leak-test
Scoring: Failure – 0%; Success – 100%.
Description: Runner2 tests whether your firewall protects a binary image of the default browser.

Schedtest
Test type: Leak-test
Scoring: Failure – 0%; Success – 100%.
Description: Schedtest checks whether your firewall allows a malicious application to schedule a task through Task Scheduler COM interface.

SSS3
Test type: General bypassing test
Scoring: Failure – 0%; Success – 100%.
Description: SSS3 initiates a system shutdown and then it checks whether your firewall protects your system until all untrusted applications are terminated.


Not a SINGLE one of these tests are what a firewall is SUPPOSED to protect against.

 

It does not really seem like a independent test if he's getting paid for his recommendations. On the other hand I'm sure he needs to get paid for his work. Who's going to spend all that time testing and not be able to make a living off your work.

 

It takes him all of a hour. if he has the right computers and the right setup. and with the money is he making that should be no problem..

the dude is to stuck on him self and the people that see those tests for the first time and believe them are seriously mislead.

 

I appreciate the list of products. It is a marginally adequate reference point. It would be better if the products listed were the current versions and were listed in alphabetic order. The other columns have no validity nor meaning and I ignore them.

 

Good, glad you knew that! IMHO pop ups or lack of popups is not the best way to test the effectiveness of a FW. In theory anyway, I could adjust the FW settings to NEVER pop up BUT that doesn't mean (at least to me) that all is well. Just that I'm not receiving any information from the FW and combining my needs with the FW.

The best FW's will ask users about an exe wanting to access to the www at least once for an exe before I allow it or not. Just because an exe, dll WANTS access doesn't mean they should have it.

As far as Matouse and money goes I don't know what the vendors and his company's contracts say.

I note that the vendors pay a royality to Matouse if a user buys their product from the Matouse site. That's the way I read the fine print there. Anybody can read it.

When the day comes if/ when Matouse recommends a product which does not have a royalty agreement that will be a signal that objectivity is gaining more ground there.

There are other testing labs of course as mentioned in the AV comparatives site where users are encouraged to read the reports from other labs. These other labs a least one of them also covers personal FW's as well.

www.icslabs.com
www.westcoastlabs.org
www.virusbtn.com

 

What is a firewall supposed to protect against?

Thanks

 

In/outbound traffic?

 

Are leak-tests considered part of outbound traffic?

Thanks

 

Leak tests are pretty much worthless, but yes, I'd consider leak tests to be under a firewall's range of duties. Stopping keyloggers and other HIPS related attacks are a different story.

 

Ah OK. Thanks. What about termination and performance and bypassing tests? And driver verifying and checking hooks?

I've heard Outpost uses user-mode hooks to pass these tests, but is useless against real malware, is it true?

Thanks

 

Yep. I think that's the case. And guess who bought PCtools ??
One free alternative maybe is gone forever. Along with TF.

The rating of the products is similar(but not the same) on testmypcsecurity, but with different order.

Strange, no PCTools firewall there or maybe is not tested.