Matousec Updates Firewall Challenge

[edd3800]

Looks like Norton Internet Security 2009 improved just about everything on NIS
http://www.matousec.com/projects/firewall-challenge/results.php

[Escalader]

{QUOTE-> Looks like Norton Internet Security 2009 improved just about everything on NIS
http://www.matousec.com/projects/firewall-challenge/results.php <-QUOTE}


Thanks for the link, but did you notice that matouse FWIW there posted

{QUOTE-> Norton Internet Security 2009 16.0.0.125 71% / 73 7 Good Not recommended <-QUOTE}

Best to stick with the recommended list I think.

[edd3800]

{QUOTE-> Thanks for the link, but did you notice that matouse FWIW there posted



Best to stick with the recommended list I think. <-QUOTE}

Yes I did I was only pointing out the improvement.Previous version Very Poor 32% (against 62 tests), the new version comes with Good protection level and final score of 71% (against 73 tests). I used Comodo for almost 2 years and after chasing down so many pop ups that turn out to be nothing. I went with another of their top picks. Only put up with that for a month and decided that a firewall/HIPS wasn't for me. I've never had a virus that wasn't caught as it was trying to infect my computer. So no more over kill for me. In the past I've used Norton 2003, then AVG both with windows firewall. Then Avira free and premium with Comodo. Then Nod with Comodo. Whatever combo I used kept my computer clean. Not once did I need the extra protection of Comodo. So in my case Norton should be just the ticket no more needless pop ups.

[wat0114]

{QUOTE->
Best to stick with the recommended list I think. <-QUOTE}

Matousec's recommendations don't really matter. If a product can favorably meet the particular requirements of a given individual, then it is an excellent product for that individual. My router would fail miserably at his poc tests, but I would not give it up for any other software security license I own, including three of my favorites: Outpost, Jetico or System Safety Monitor. He rates the latter two products, not coincidentally, "not recommended". I don't care. IMO they are "excellent". For my needs they provide the security I'm looking for, and then some.

[Fajo]

the way he test is so flawed that a it could not hold up to a fly. ::)

[entropism]

A firewall test that doesn't test the capabilities of a firewall. YAY!
A firewall test that tests software that... isn't a firewall? WTF?

Matsousec was who I turned to WAY back in the day, but it's just a joke now that I know better. If you run a separate HIPS, pretty much every *firewall* on that list would be perfect. The fact that he tests HIPS programs that don't even include a firewall is just hilarious.

[edd3800]

I'm not an advance user. I did read the tests he performed and to me it seemed like it was much more than just a firewall test. In the future if I fill I need more protection I could always add TF. Don't get me wrong Comodo was great. I don't think anything went on in my system without Comodo giving me a pop up.
If I did get infected I'm sure Comodo would have alerted me. The problem wasn't Comodo it was me looking at the alert and making a Homer Simpson comment to myself.

[nhamilton]

if it means anything according to the Matousec results the PCtools firewall went from 6% in version 3 to now 85% in version 4.

[GES/POR]

1 have 1 issue only with Matousec's test and that's the reccomendations. How is it that products rated as good are NOT RECOMMENDED - beats the .... out of me!

[entropism]

The ones he recommends are the ones he has a referral link to, so he makes money on every purchase. According to the affiliate plans from Outpost, up to 50% of the sale is commission.

[doktornotor]

{QUOTE-> The ones he recommends are the ones he has a referral link to, so he makes money on every purchase. According to the affiliate plans from Outpost, up to 50% of the sale is commission. <-QUOTE}

Yup, exactly. Vendors who don't agree upon such pretty weird (for "independent" testing area) agreement or who offer free products (Comodo) do not get "recommendations" no matter what their result is. Matousec is about money and nothing else. :thumbd:

[Someone]

{QUOTE-> the way he test is so flawed that a it could not hold up to a fly. ::) <-QUOTE}
How is the test flawed?

{QUOTE-> A firewall test that tests software that... isn't a firewall? WTF? <-QUOTE}
Don't those HIPS have a outbound firewall?

Thanks

[doktornotor]

{QUOTE-> How is the test flawed?
Don't those HIPS have a outbound firewall?
<-QUOTE}

He's not testing firewalls (http://www.wilderssecurity.com/showthread.php?p=1318898#post1318898) in the first place. The whole "firewall challenge" is a misnomer.

[Someone]

{QUOTE-> He's not testing firewalls (http://www.wilderssecurity.com/showthread.php?p=1318898#post1318898) in the first place. The whole "firewall challenge" is a misnomer. <-QUOTE}
Thanks for the explanation. Are leak tests considered something an outbound firewall without HIPS should block? Wouldn't the driver and OS crashes be related to a firewall?

[entropism]

Take a look at what some of the tests cover, they're all over the place. Why should a firewall have to detect a keylogger? let's take a look at his tests at level 7:

BITStest
Test type: Leak-test
Scoring: Failure – 0%; Success – 100%.
Description: BITStest checks if it is possible to bypass your firewall by using Background Intelligent Transfer Service (BITS) through Background Copy Manager COM interface.

Keylog5
Test type: Spying test
Scoring: Failure – 0%; Success – 100%.
Description: Keylog5 repeatedly attaches the keyboard input of the foreground windows' process to be able to sniff the user's input of that process.

Keylog6
Test type: Spying test
Scoring: Failure – 0%; Success – 100%.
Description: Keylog6 registers a raw input device to be able to monitor user's keystrokes.

OSfwbypass
Test type: Leak-test
Scoring: Failure – 0%; Success – 100%.
Description: OSfwbypass tests whether your firewall is able to deny an attempt to display and execute contents of a malicious HTML page. This attempt is performed by a special API.

Runner2
Test type: Leak-test
Scoring: Failure – 0%; Success – 100%.
Description: Runner2 tests whether your firewall protects a binary image of the default browser.

Schedtest
Test type: Leak-test
Scoring: Failure – 0%; Success – 100%.
Description: Schedtest checks whether your firewall allows a malicious application to schedule a task through Task Scheduler COM interface.

SSS3
Test type: General bypassing test
Scoring: Failure – 0%; Success – 100%.
Description: SSS3 initiates a system shutdown and then it checks whether your firewall protects your system until all untrusted applications are terminated.


Not a SINGLE one of these tests are what a firewall is SUPPOSED to protect against.

[edd3800]

{QUOTE-> Yup, exactly. Vendors who don't agree upon such pretty weird (for "independent" testing area) agreement or who offer free products (Comodo) do not get "recommendations" no matter what their result is. Matousec is about money and nothing else. :thumbd: <-QUOTE}

It does not really seem like a independent test if he's getting paid for his recommendations. On the other hand I'm sure he needs to get paid for his work. Who's going to spend all that time testing and not be able to make a living off your work.

[Fajo]

{QUOTE-> It does not really seem like a independent test if he's getting paid for his recommendations. On the other hand I'm sure he needs to get paid for his work. Who's going to spend all that time testing and not be able to make a living off your work. <-QUOTE}

It takes him all of a hour. if he has the right computers and the right setup. and with the money is he making that should be no problem..

the dude is to stuck on him self and the people that see those tests for the first time and believe them are seriously mislead. :dry:

[AKAJohnDoe]

I appreciate the list of products. It is a marginally adequate reference point. It would be better if the products listed were the current versions and were listed in alphabetic order. The other columns have no validity nor meaning and I ignore them.

[Escalader]

{QUOTE-> Yes I did I was only pointing out the improvement.Previous version Very Poor 32% (against 62 tests), the new version comes with Good protection level and final score of 71% (against 73 tests). I used Comodo for almost 2 years and after chasing down so many pop ups that turn out to be nothing. I went with another of their top picks. Only put up with that for a month and decided that a firewall/HIPS wasn't for me. I've never had a virus that wasn't caught as it was trying to infect my computer. So no more over kill for me. In the past I've used Norton 2003, then AVG both with windows firewall. Then Avira free and premium with Comodo. Then Nod with Comodo. Whatever combo I used kept my computer clean. Not once did I need the extra protection of Comodo. So in my case Norton should be just the ticket no more needless pop ups. <-QUOTE}



Good, glad you knew that! IMHO pop ups or lack of popups is not the best way to test the effectiveness of a FW. In theory anyway, I could adjust the FW settings to NEVER pop up BUT that doesn't mean (at least to me) that all is well. Just that I'm not receiving any information from the FW and combining my needs with the FW.

The best FW's will ask users about an exe wanting to access to the www at least once for an exe before I allow it or not. Just because an exe, dll WANTS access doesn't mean they should have it.

As far as Matouse and money goes I don't know what the vendors and his company's contracts say.

I note that the vendors pay a royality to Matouse if a user buys their product from the Matouse site. That's the way I read the fine print there. Anybody can read it.

When the day comes if/ when Matouse recommends a product which does not have a royalty agreement that will be a signal that objectivity is gaining more ground there.

There are other testing labs of course as mentioned in the AV comparatives site where users are encouraged to read the reports from other labs. These other labs a least one of them also covers personal FW's as well.

www.icslabs.com
www.westcoastlabs.org
www.virusbtn.com

[Someone]

{QUOTE-> Not a SINGLE one of these tests are what a firewall is SUPPOSED to protect against. <-QUOTE}
What is a firewall supposed to protect against?

Thanks

[entropism]

In/outbound traffic?

[Someone]

{QUOTE-> In/outbound traffic? <-QUOTE}
Are leak-tests considered part of outbound traffic?

Thanks

[entropism]

Leak tests are pretty much worthless, but yes, I'd consider leak tests to be under a firewall's range of duties. Stopping keyloggers and other HIPS related attacks are a different story.

[Someone]

{QUOTE-> Leak tests are pretty much worthless, but yes, I'd consider leak tests to be under a firewall's range of duties. Stopping keyloggers and other HIPS related attacks are a different story. <-QUOTE}
Ah OK. Thanks. What about termination and performance and bypassing tests? And driver verifying and checking hooks?

I've heard Outpost uses user-mode hooks to pass these tests, but is useless against real malware, is it true?

Thanks

[renegade08]

{QUOTE-> if it means anything according to the Matousec results the PCtools firewall went from 6% in version 3 to now 85% in version 4. <-QUOTE}


Yep. I think that's the case. And guess:o who bought PCtools:'( ??
One free alternative maybe is gone forever. Along with TF.

The rating of the products is similar(but not the same) on testmypcsecurity (http://www.testmypcsecurity.com/view_results_xp.html), but with different order.

Strange, no PCTools firewall there or maybe is not tested.

[Escalader]

{QUOTE-> What is a firewall supposed to protect against?

Thanks <-QUOTE}

Hello:

For excellent easy learning on basic FW questions please refer to these 2 excellent stickies:

http://www.wilderssecurity.com/showthread.php?t=142036
http://www.wilderssecurity.com/showthread.php?t=24415

[Escalader]

{QUOTE-> In/outbound traffic? <-QUOTE}


Have a read at:

http://www.wilderssecurity.com/showthread.php?t=142036
http://www.wilderssecurity.com/showthread.php?t=24415

[Escalader]

{QUOTE-> Are leak-tests considered part of outbound traffic?

Thanks <-QUOTE}


All test definitions for FW's as defined by those doing the testing in this case Matousec determines the criteria that they use. Other Labs and testers use different criteria.

However, the common meanings for outbound testing is "leaking" or calling home.

Some view FW's as being ONLY inbound traffic managment others ( me included) want THEIR FW's to manage both.

Every user IMHO should have their own Security Policy written down otherwise we run the risk of being blown from pillar to post with every new product or FUD based post from "experts".

[entropism]

{QUOTE-> Have a read at:

http://www.wilderssecurity.com/showthread.php?t=142036
http://www.wilderssecurity.com/showthread.php?t=24415 <-QUOTE}

I wasn't questioning my answer, I know what a firewall is there for. ::)

[Escalader]

{QUOTE-> I wasn't questioning my answer, I know what a firewall is there for. ::) <-QUOTE}


That's good! 8)

[Someone]

{QUOTE-> Hello:

For excellent easy learning on basic FW questions please refer to these 2 excellent stickies:

http://www.wilderssecurity.com/showthread.php?t=142036
http://www.wilderssecurity.com/showthread.php?t=24415 <-QUOTE}
Thanks for the links :thumb:, I've just read them.

[Escalader]

{QUOTE-> Thanks for the links :thumb:, I've just read them. <-QUOTE}


That's good, come over to the Hints on OA Learning Thread and browse around lots of interesting FAQ!

[Espresso]

{QUOTE-> I appreciate the list of products. It is a marginally adequate reference point. It would be better if the products listed were the current versions and were listed in alphabetic order. The other columns have no validity nor meaning and I ignore them. <-QUOTE}

It would be useful if the site allowed setting up a comparison table with the tests of your own choice. Then you can use your own definition of "firewall" to define the applicable tests with a resulting score that actually means something.

Of course, he has no incoming packet tests so it would still be worthless to some people.

[AKAJohnDoe]

{QUOTE-> It would be useful if the site allowed setting up a comparison table with the tests of your own choice. <-QUOTE}
That is exactly the way I feel about Consumer Reports as well (and suggested to them years ago). If the criteria and weighting of each reviewed item could be extracted into a spreadsheet, then the importance (weighting) of each criteria could be individually determined and additional criteria added.

Although, I eventually came to the realization that I could do that myself without them.

[fax]

{QUOTE-> 2008-09-28:
New results have been published for:
BitDefender Internet Security 2009 12.0.10.2
McAfee Internet Security 2009 10.0.209
ZoneAlarm Pro 2009 8.0.020.000

Among the 2009 solutions we have tested today, only ZoneAlarm Pro is worth mentioning. Its previously tested version 7 scored only 63% (against 62 tests). The new version has been improved so that it passed all the levels with the final score of 86%, which is a Very good result.

BitDefender Internet Security 2009 and McAfee Internet Security 2009 failed in the tests completely. BitDefender 2008 scored 4% (against 70 tests) and its 2009 version scored 7% (against 73 tests) remaining on None Protection level. McAfee version 2009 finished with the final score of 12% (against 73 tests), its previously tested version scored 6% (against 70 tests). <-QUOTE}

For the Matousec fans... the list of firewall tested has been updated.

Cheers,
Fax