Matousec - Major changes in Firewall Challenge

Discussion in 'other firewalls' started by Einsturzende, Nov 7, 2008.

Thread Status:
Not open for further replies.
  1. Einsturzende

    Einsturzende Registered Member

    Joined:
    Apr 14, 2008
    Posts:
    390
    Location:
    neubauten
    "Firewall Challenge moves forward. Several noticeable changes and improvements have been implemented:
    1) We are leaving Windows XP Service Pack 2 platform. All new tests will be performed on Windows XP Service Pack 3. If a product is not compatible with Windows XP Service Pack 3, we may make an exception if our visitors are interested in this product.
    2) We are raising the bar of Firewall Challenge. We have implemented several new tests to SSTS, which we also use in Firewall Challenge.
    3) We have also improved some methods used by various tests, mostly termination tests, hence their power has increased.
    4) Another thing that makes the challenge tougher for the tested products is a change of the scoring of tests SSS, SSS2 and SSS3. Newly, we penalize inabilities to block an unwanted user logout and a system shutdown.
    5) We are leaving two tests PerfTCP and PerfUDP, which caused more problems than benefits. Mixing the performance and security tests turned out not to be a good idea. These tests will remain in the system only because of the products that were tested with them. New challenges will be performed without these two tests.
    6) We are changing the testing method based on Driver Verifier. We will not perform subtests for each of the options that Driver Verifier supports anymore. A single combined Driver Verifier test will be performed. This change would make it hard for the products to pass the level 9, hence we have added one more test to the level 9 and changed the score limit for this level from 75% to 50%."


    "Changelog:

    * 2008-11-07: Thirteen new tests have been added to the suite, two tests have been removed, several tests have been improved, some changes have been done to the licence agreement. Kernel1 has been added to the Level 3; Kernel1b has been added to the Level 4; Kernel2, Kernel3 and Crash4 have been added to the Level 5; Kernel4, Crash5 and Crash6 have been added to the Level 6; FireHole2 and Kill12 have been added to the Level 7; Kernel4b and Kernel5 have been added to the Level 8; Crash7 has been added to the Level 9. PerfTCP and PerfUDP have been removed from the suite – mixing the performance and security tests turned out not to be a good idea. The current number of the tests in the suite is 81."

    MY EDIT:
    I found Rootkit (driver loading) tests particularly very interesting, also I tested KIS 2009 against "kernel" tests and KIS passed all of them :)

    EDIT1: If admins or/and mods find that this should be posted at "other anti-malware software" please move it there :)
     
    Last edited: Nov 7, 2008
  2. Einsturzende

    Einsturzende Registered Member

    Joined:
    Apr 14, 2008
    Posts:
    390
    Location:
    neubauten
    Im curently testing OA3 (paid trial) on virtual machine and it failed "kernel4b", can anybody confirm?
     
  3. Fuzzfas

    Fuzzfas Registered Member

    Joined:
    Jun 24, 2007
    Posts:
    2,753
    Damn! These were the most interesting tests for me!
     
  4. MaB69

    MaB69 Registered Member

    Joined:
    Dec 9, 2005
    Posts:
    540
    Location:
    Paris
    Hi,

    I totally agree with u : the only one that truly test firewall features

    Regards,

    MaB
     
  5. alex_s

    alex_s Registered Member

    Joined:
    Aug 13, 2007
    Posts:
    1,251
    I really like Vista :)


    D:\Download\ssts\bin\Level 8>kernel4b.exe
    Security Software Testing Suite - Kernel4b
    Copyright by Matousec - Transparent security
    http://www.matousec.com/


    ERROR: Unable to open registry key "HKLM\SYSTEM\CurrentControlSet\Services\audst
    ub".
    Error code: 2

    YOUR SYSTEM PASSED THE TEST!
     
  6. alex_s

    alex_s Registered Member

    Joined:
    Aug 13, 2007
    Posts:
    1,251
    I can send them to you from the old set, if you need them :)
     
  7. Einsturzende

    Einsturzende Registered Member

    Joined:
    Apr 14, 2008
    Posts:
    390
    Location:
    neubauten
    So on Vista test gives error... BTW I would really like resources addition for OA like in some other appz that type, for now OA is too stiff and depending on developer decisions.
     
  8. MaB69

    MaB69 Registered Member

    Joined:
    Dec 9, 2005
    Posts:
    540
    Location:
    Paris
    Hi,

    Thanks Alex, they are already in my Leaktest folder ;)

    Regards,

    MaB
     
  9. alex_s

    alex_s Registered Member

    Joined:
    Aug 13, 2007
    Posts:
    1,251
    Yes, you are right, and this is intentional way. OA is positioned as "as litttle user intereaction as possible" product and looks for the way to decreas it. Most users do not need too many options becase they do not know what to do with them. As for me I'm sure true security must be completely silent and doesn't need any tweaking. This is, of course, unreachable ideal goal, though it is the only direction "security for people" product can follow to be successful.

    PS. And yes, On Vista many XP tests and exploints fail without any additional security, even using admin account with UAC off. I was forced to move to Vista because my new laptop had it preinstalled and it was said that the drivers I need to enjoy the full power of my new hardware exist only for Vista. It took 3-4 weeks for me to start not to sorry for XP :)
     
  10. doktornotor

    doktornotor Registered Member

    Joined:
    Jul 19, 2008
    Posts:
    2,047
    Sooo... Now the only major change remaining is to rename the test to something that actually describes what they are doing. :rolleyes:
     
Loading...
Thread Status:
Not open for further replies.