Matousec: Proactive Security Challenge 64 (bits)

hell, i tried to run their suit

 

the 2013 16.30.0.1843 version

 

http://www.matousec.com/projects/proactive-security-challenge-64/results.php

 

http://www.matousec.com/projects/proactive-security-challenge-64/results.php

 

http://www.matousec.com/projects/proactive-security-challenge-64/

 

See https://www.wilderssecurity.com/showpost.php?p=2336261&postcount=20.

 

Examples of real-world malware bypassing outbound protection in firewalls: https://www.wilderssecurity.com/showpost.php?p=2336361&postcount=22.

 

As guest said, this is a test of HIPS, and also spying/leaking.

From http://www.matousec.com/projects/proactive-security-challenge-64/:

You should use caution when using these tests on a behavioral blocker. A behavior blocker may notice Action 1 and Action 2, but only alert after seeing both of those actions performed by the same program.

IMHO the testing level methodology used is demonstrably awful. For example, consider ZoneAlarm Free AF. On test #1, it scored 90%; on test #6 it scored 40%. If the tests in test #6 had been performed first, then ZoneAlarm Free AF would have flunked the first level and ended up with a very low overall score.

 

Because of the arbitrary methodology of the cut-off points of their levels their final results seem misleading to me.

 

I agree. Levels might be ok if the first levels are the easiest, but I haven't seen evidence that that is the case.

 

Follow-up to post #58:

 

Here's an alphabetized list of all the techniques used:

binary planting
code injection
COM interface exploitation
DDE exploitation
direct network access
disk location exploitation
DLL injection
file/directory manipulation
graphics API exploitation
in-process data substitution
indirect network access
keyboard API exploitation
network API exploitation
network management API exploitation
parent process control bypassing
registry key/value manipulation
registry location exploitation
remote process handles manipulation
remote process manipulation
remote process memory manipulation
remote thread creation
remote thread manipulation
system object manipulation
system service exploitation
trusted process manipulation
windows clipboard API exploitation
Windows Filtering Platform API exploitation
Windows Management Instrumentation API exploitation
windows messages exploitation
windows/event hooking exploitation

 

Here is a good introduction to leak testing from 2007. I haven't found a good newer reference yet.

 

Wow, still the same useless tests. If you have enough money your application can be retested again and again (until it's number one).

 

Yeap, main weaknesses of these tests have been discussed at lenght many times in the past 4-5 years. There are several long threads about it including comments from experts. So, good to make a quick search on "matousec" to find them to avoid opening this can of worms again and again.