Matousec - Major changes in Firewall Challenge

"Firewall Challenge moves forward. Several noticeable changes and improvements have been implemented:
1) We are leaving Windows XP Service Pack 2 platform. All new tests will be performed on Windows XP Service Pack 3. If a product is not compatible with Windows XP Service Pack 3, we may make an exception if our visitors are interested in this product.
2) We are raising the bar of Firewall Challenge. We have implemented several new tests to SSTS, which we also use in Firewall Challenge.
3) We have also improved some methods used by various tests, mostly termination tests, hence their power has increased.
4) Another thing that makes the challenge tougher for the tested products is a change of the scoring of tests SSS, SSS2 and SSS3. Newly, we penalize inabilities to block an unwanted user logout and a system shutdown.
5) We are leaving two tests PerfTCP and PerfUDP, which caused more problems than benefits. Mixing the performance and security tests turned out not to be a good idea. These tests will remain in the system only because of the products that were tested with them. New challenges will be performed without these two tests.
6) We are changing the testing method based on Driver Verifier. We will not perform subtests for each of the options that Driver Verifier supports anymore. A single combined Driver Verifier test will be performed. This change would make it hard for the products to pass the level 9, hence we have added one more test to the level 9 and changed the score limit for this level from 75% to 50%."

"Changelog:

* 2008-11-07: Thirteen new tests have been added to the suite, two tests have been removed, several tests have been improved, some changes have been done to the licence agreement. Kernel1 has been added to the Level 3; Kernel1b has been added to the Level 4; Kernel2, Kernel3 and Crash4 have been added to the Level 5; Kernel4, Crash5 and Crash6 have been added to the Level 6; FireHole2 and Kill12 have been added to the Level 7; Kernel4b and Kernel5 have been added to the Level 8; Crash7 has been added to the Level 9. PerfTCP and PerfUDP have been removed from the suite – mixing the performance and security tests turned out not to be a good idea. The current number of the tests in the suite is 81."

MY EDIT:
I found Rootkit (driver loading) tests particularly very interesting, also I tested KIS 2009 against "kernel" tests and KIS passed all of them

EDIT1: If admins or/and mods find that this should be posted at "other anti-malware software" please move it there

 

Im curently testing OA3 (paid trial) on virtual machine and it failed "kernel4b", can anybody confirm?

 

Damn! These were the most interesting tests for me!

 

Hi,

I totally agree with u : the only one that truly test firewall features

Regards,

MaB

 

I really like Vista


D:\Download\ssts\bin\Level 8>kernel4b.exe
Security Software Testing Suite - Kernel4b
Copyright by Matousec - Transparent security
http://www.matousec.com/


ERROR: Unable to open registry key "HKLM\SYSTEM\CurrentControlSet\Services\audst
ub".
Error code: 2

YOUR SYSTEM PASSED THE TEST!

 

I can send them to you from the old set, if you need them

 

So on Vista test gives error... BTW I would really like resources addition for OA like in some other appz that type, for now OA is too stiff and depending on developer decisions.

 

Hi,

Thanks Alex, they are already in my Leaktest folder

Regards,

MaB

 

Yes, you are right, and this is intentional way. OA is positioned as "as litttle user intereaction as possible" product and looks for the way to decreas it. Most users do not need too many options becase they do not know what to do with them. As for me I'm sure true security must be completely silent and doesn't need any tweaking. This is, of course, unreachable ideal goal, though it is the only direction "security for people" product can follow to be successful.

PS. And yes, On Vista many XP tests and exploints fail without any additional security, even using admin account with UAC off. I was forced to move to Vista because my new laptop had it preinstalled and it was said that the drivers I need to enjoy the full power of my new hardware exist only for Vista. It took 3-4 weeks for me to start not to sorry for XP

 

Sooo... Now the only major change remaining is to rename the test to something that actually describes what they are doing.