Matousec Updates Firewall Challenge

Discussion in 'other firewalls' started by edd3800, Sep 18, 2008.

Thread Status:
Not open for further replies.
  1. edd3800

    edd3800 Registered Member

    Joined:
    Nov 11, 2007
    Posts:
    48
  2. Escalader

    Escalader Registered Member

    Joined:
    Dec 12, 2005
    Posts:
    3,710
    Location:
    Land of the Mooses

    Thanks for the link, but did you notice that matouse FWIW there posted

    Best to stick with the recommended list I think.
     
  3. edd3800

    edd3800 Registered Member

    Joined:
    Nov 11, 2007
    Posts:
    48
    Yes I did I was only pointing out the improvement.Previous version Very Poor 32% (against 62 tests), the new version comes with Good protection level and final score of 71% (against 73 tests). I used Comodo for almost 2 years and after chasing down so many pop ups that turn out to be nothing. I went with another of their top picks. Only put up with that for a month and decided that a firewall/HIPS wasn't for me. I've never had a virus that wasn't caught as it was trying to infect my computer. So no more over kill for me. In the past I've used Norton 2003, then AVG both with windows firewall. Then Avira free and premium with Comodo. Then Nod with Comodo. Whatever combo I used kept my computer clean. Not once did I need the extra protection of Comodo. So in my case Norton should be just the ticket no more needless pop ups.
     
  4. wat0114

    wat0114 Guest

    Matousec's recommendations don't really matter. If a product can favorably meet the particular requirements of a given individual, then it is an excellent product for that individual. My router would fail miserably at his poc tests, but I would not give it up for any other software security license I own, including three of my favorites: Outpost, Jetico or System Safety Monitor. He rates the latter two products, not coincidentally, "not recommended". I don't care. IMO they are "excellent". For my needs they provide the security I'm looking for, and then some.
     
  5. Fajo

    Fajo Registered Member

    Joined:
    Jun 13, 2008
    Posts:
    1,812
    the way he test is so flawed that a it could not hold up to a fly. :rolleyes:
     
  6. entropism

    entropism Registered Member

    Joined:
    Dec 9, 2004
    Posts:
    333
    A firewall test that doesn't test the capabilities of a firewall. YAY!
    A firewall test that tests software that... isn't a firewall? WTF?

    Matsousec was who I turned to WAY back in the day, but it's just a joke now that I know better. If you run a separate HIPS, pretty much every *firewall* on that list would be perfect. The fact that he tests HIPS programs that don't even include a firewall is just hilarious.
     
  7. edd3800

    edd3800 Registered Member

    Joined:
    Nov 11, 2007
    Posts:
    48
    I'm not an advance user. I did read the tests he performed and to me it seemed like it was much more than just a firewall test. In the future if I fill I need more protection I could always add TF. Don't get me wrong Comodo was great. I don't think anything went on in my system without Comodo giving me a pop up.
    If I did get infected I'm sure Comodo would have alerted me. The problem wasn't Comodo it was me looking at the alert and making a Homer Simpson comment to myself.
     
  8. nhamilton

    nhamilton Registered Member

    Joined:
    Jul 31, 2007
    Posts:
    61
    if it means anything according to the Matousec results the PCtools firewall went from 6% in version 3 to now 85% in version 4.
     
  9. GES/POR

    GES/POR Registered Member

    Joined:
    Nov 26, 2006
    Posts:
    1,490
    Location:
    Armacham
    1 have 1 issue only with Matousec's test and that's the reccomendations. How is it that products rated as good are NOT RECOMMENDED - beats the .... out of me!
     
  10. entropism

    entropism Registered Member

    Joined:
    Dec 9, 2004
    Posts:
    333
    The ones he recommends are the ones he has a referral link to, so he makes money on every purchase. According to the affiliate plans from Outpost, up to 50% of the sale is commission.
     
  11. doktornotor

    doktornotor Registered Member

    Joined:
    Jul 19, 2008
    Posts:
    2,047
    Yup, exactly. Vendors who don't agree upon such pretty weird (for "independent" testing area) agreement or who offer free products (Comodo) do not get "recommendations" no matter what their result is. Matousec is about money and nothing else. :thumbd:
     
  12. Someone

    Someone Registered Member

    Joined:
    Jan 18, 2008
    Posts:
    1,106
    How is the test flawed?

    Don't those HIPS have a outbound firewall?

    Thanks
     
  13. doktornotor

    doktornotor Registered Member

    Joined:
    Jul 19, 2008
    Posts:
    2,047
    He's not testing firewalls in the first place. The whole "firewall challenge" is a misnomer.
     
  14. Someone

    Someone Registered Member

    Joined:
    Jan 18, 2008
    Posts:
    1,106
    Thanks for the explanation. Are leak tests considered something an outbound firewall without HIPS should block? Wouldn't the driver and OS crashes be related to a firewall?
     
  15. entropism

    entropism Registered Member

    Joined:
    Dec 9, 2004
    Posts:
    333
    Take a look at what some of the tests cover, they're all over the place. Why should a firewall have to detect a keylogger? let's take a look at his tests at level 7:

    BITStest
    Test type: Leak-test
    Scoring: Failure – 0%; Success – 100%.
    Description: BITStest checks if it is possible to bypass your firewall by using Background Intelligent Transfer Service (BITS) through Background Copy Manager COM interface.

    Keylog5
    Test type: Spying test
    Scoring: Failure – 0%; Success – 100%.
    Description: Keylog5 repeatedly attaches the keyboard input of the foreground windows' process to be able to sniff the user's input of that process.

    Keylog6
    Test type: Spying test
    Scoring: Failure – 0%; Success – 100%.
    Description: Keylog6 registers a raw input device to be able to monitor user's keystrokes.

    OSfwbypass
    Test type: Leak-test
    Scoring: Failure – 0%; Success – 100%.
    Description: OSfwbypass tests whether your firewall is able to deny an attempt to display and execute contents of a malicious HTML page. This attempt is performed by a special API.

    Runner2
    Test type: Leak-test
    Scoring: Failure – 0%; Success – 100%.
    Description: Runner2 tests whether your firewall protects a binary image of the default browser.

    Schedtest
    Test type: Leak-test
    Scoring: Failure – 0%; Success – 100%.
    Description: Schedtest checks whether your firewall allows a malicious application to schedule a task through Task Scheduler COM interface.

    SSS3
    Test type: General bypassing test
    Scoring: Failure – 0%; Success – 100%.
    Description: SSS3 initiates a system shutdown and then it checks whether your firewall protects your system until all untrusted applications are terminated.


    Not a SINGLE one of these tests are what a firewall is SUPPOSED to protect against.
     
  16. edd3800

    edd3800 Registered Member

    Joined:
    Nov 11, 2007
    Posts:
    48
    It does not really seem like a independent test if he's getting paid for his recommendations. On the other hand I'm sure he needs to get paid for his work. Who's going to spend all that time testing and not be able to make a living off your work.
     
  17. Fajo

    Fajo Registered Member

    Joined:
    Jun 13, 2008
    Posts:
    1,812
    It takes him all of a hour. if he has the right computers and the right setup. and with the money is he making that should be no problem..

    the dude is to stuck on him self and the people that see those tests for the first time and believe them are seriously mislead. :cautious:
     
  18. AKAJohnDoe

    AKAJohnDoe Registered Member

    Joined:
    Sep 26, 2007
    Posts:
    989
    Location:
    127.0.0.1
    I appreciate the list of products. It is a marginally adequate reference point. It would be better if the products listed were the current versions and were listed in alphabetic order. The other columns have no validity nor meaning and I ignore them.
     
  19. Escalader

    Escalader Registered Member

    Joined:
    Dec 12, 2005
    Posts:
    3,710
    Location:
    Land of the Mooses


    Good, glad you knew that! IMHO pop ups or lack of popups is not the best way to test the effectiveness of a FW. In theory anyway, I could adjust the FW settings to NEVER pop up BUT that doesn't mean (at least to me) that all is well. Just that I'm not receiving any information from the FW and combining my needs with the FW.

    The best FW's will ask users about an exe wanting to access to the www at least once for an exe before I allow it or not. Just because an exe, dll WANTS access doesn't mean they should have it.

    As far as Matouse and money goes I don't know what the vendors and his company's contracts say.

    I note that the vendors pay a royality to Matouse if a user buys their product from the Matouse site. That's the way I read the fine print there. Anybody can read it.

    When the day comes if/ when Matouse recommends a product which does not have a royalty agreement that will be a signal that objectivity is gaining more ground there.

    There are other testing labs of course as mentioned in the AV comparatives site where users are encouraged to read the reports from other labs. These other labs a least one of them also covers personal FW's as well.

    www.icslabs.com
    www.westcoastlabs.org
    www.virusbtn.com
     
  20. Someone

    Someone Registered Member

    Joined:
    Jan 18, 2008
    Posts:
    1,106
    What is a firewall supposed to protect against?

    Thanks
     
  21. entropism

    entropism Registered Member

    Joined:
    Dec 9, 2004
    Posts:
    333
    In/outbound traffic?
     
  22. Someone

    Someone Registered Member

    Joined:
    Jan 18, 2008
    Posts:
    1,106
    Are leak-tests considered part of outbound traffic?

    Thanks
     
  23. entropism

    entropism Registered Member

    Joined:
    Dec 9, 2004
    Posts:
    333
    Leak tests are pretty much worthless, but yes, I'd consider leak tests to be under a firewall's range of duties. Stopping keyloggers and other HIPS related attacks are a different story.
     
  24. Someone

    Someone Registered Member

    Joined:
    Jan 18, 2008
    Posts:
    1,106
    Ah OK. Thanks. What about termination and performance and bypassing tests? And driver verifying and checking hooks?

    I've heard Outpost uses user-mode hooks to pass these tests, but is useless against real malware, is it true?

    Thanks
     
    Last edited: Sep 20, 2008
  25. renegade08

    renegade08 Registered Member

    Joined:
    Aug 26, 2008
    Posts:
    431

    Yep. I think that's the case. And guess:eek: who bought PCtools:'( ??
    One free alternative maybe is gone forever. Along with TF.

    The rating of the products is similar(but not the same) on testmypcsecurity, but with different order.

    Strange, no PCTools firewall there or maybe is not tested.
     
Loading...
Thread Status:
Not open for further replies.