Matousec: Proactive Security Challenge 64 (bits)

Proactive Security Challenge 64 (2011/12/01 11:26)

Proactive Security Challenge 64 starts and it is not just an update to another platform.

Yes, one of the changes is that the 32-bit Windows XP SP3 platform used in the original Proactive Security Challenge is replaced with 64-bit Windows 7 SP1 platform, but that is not the only change. Naturally, Proactive Security Challenge 64 uses Security Software Testing Suite 64 that we presented last time. The new suite comes with lots of improvements, new and harder tests. The structure of levels has been carefully redesigned so that in Proactive Security Challenge 64 the very bad products will still score very low, sometimes possibly little better than in the original Proactive Security Challenge; the products of medium quality should score better, sometimes much better; and the very good and excellent products will have it much harder to achieve the perfect score. So, for most of the products, the challenge is easier and will allow better scoring, but for the top products it will be harder to achieve the same score as before, at least at the beginning. In other words, the new suite and the new system of levels is better balanced than before.

We start with just four products, but in a couple of weeks there will be more. The results of Jetico Personal Firewall confirm that it is not that easy to get really high score. The performance of the new ESET Smart Security 5 suggests that its vendor might want to join the club but it will take some time before its engine matures. The results of McAfee Total Protection and Panda Global Protection are not surprising. Unless a big change happens here, they will be on the bottom of the result table. See the first results of Proactive Security Challenge 64 yourself.

permalink

 

It's kinda funny they rate ESET SS very poor, yet display a prominent advertisement for it to the left of the results

Jetico has not been updated by the developer since last December

 

It's more relevant, I guess. Where's all the other participants?

 

"We start with just four products, but in a couple of weeks there will be more." ~Matousec

 

The tests themselves are good but it's too bad they still keep this absurd level structure, as it destroys all objectivity. Have to to tests myself if I want accurate and true results.

 

Quite agree, the level structure is just silly.

 

Yeah, I don't see the point in the level structure either. Malware doesn't come in levels... you don't need one infection before you can get another.

 

Indeed. It's as if AV-Comparatives don't test a certain antivirus program against trojans unless it scores well against viruses first. It makes absolutely no sense from a logical and/or practical standpoint. It does make sense from a marketing standpoint, though, and that's all that matters nowadays. Unfortunately.

 

Latest news

2011-12-21: New results have been published for:
AVG Internet Security 2012.0.1890
avast! Internet Security 6.0.1367
Avira Internet Security 2012 12.0.0.832
PC Tools Internet Security 2012 9.0.0.898

This time we wanted to give you a product that scores better than others, possibly attacks the 80% bar. This is why we included PC Tools Internet Security in this update. It was a big and unpleasant surprise that its performance in Proactive Security Challenge 64 was extremely poor. If you remember its excellent results from Proactive Security Challenge, you must be surprised too. On 64-bit Windows 7 SP1 platform, the tested version of PC Tools Internet Security 2012 succeeded against just a few tests and its final score is as low as 6 %. We hope that the related issues in its protection will be solved quickly and that PC Tools Internet Security will strike back among the top products soon.

Since PC Tools Internet Security performed poorly, it was avast! Internet Security that finished with the best score among the products tested this time. Yet still its performance is not good enough. It offers only a basic protection against some of the attacking techniques leaving malware plenty of ways to bypass its protection. Its final score is 15 %, still deep in the red zone.

AVG Internet Security and Avira Internet Security remain consistent in their poor results. Nothing new on this field – 3% score for AVG, 4% for Avira. Both products pass only the most trivial tests.

Source: http://www.matousec.com/projects/proactive-security-challenge-64/#latest-news

 

Or as in a real-world translation;
On Win7 SP1 x64, PC Tools failed at test nr. 7 out of a hundred, so we decided to stop testing and thus give it a score of 6% effectiveness because it didn't achieve any scores at tests 7 to 100.
Meh, bizarro world level structure testing indeed.

 

Latest news

2011-01-13: New results have been published for:
Comodo Internet Security Premium 5.9.219863.2196
F-Secure Internet Security 2012 12.44.110
G Data TotalSecurity 2012 22.1.0.0
Privatefirewall 7.0.25.5

Comodo Internet Security did it again! There were few tests that it failed this time and it is probably just a matter of time for this product to reach the perfect score in our challenge again. Comodo Internet Security outclassed all products that have been tested in Proactive Security Challenge 64 so far. With 94% score its position on the top of the table is stronger than ever before. Congratulations!

Privatefirewall looked very promising on lower levels, but harder tests on higher levels did not allow this popular free product to get to the top. 56% score currently means 3rd place in Proactive Security Challenge 64. Privatefirewall's core is solid, its self-protection is almost perfect and the autorun protection control is very good. Having long term experience with this product, we expect that Privatefirewall will be improved in the future and will strengthen its position on the top of the result table.

F-Secure Internet Security 2012 and G Data TotalSecurity 2012 did not surprise us. Although G Data TotalSecurity performed little better than F-Secure Internet Security, its final score is as low as 5 %. The implementation of many of its features is incomplete – for example the autorun control or the self-defense features. F-Secure Internet Security then passed even less tests and finished with 3% score.

Source:
http://www.matousec.com/projects/proactive-security-challenge-64/#latest-news
http://www.matousec.com/projects/proactive-security-challenge-64/results.php

 

That's odd. I tested Comodo Firewall (v5.8 though) myself and got a significantly lower score than that. Also, results were different with the enhanced protection disabled/enabled. Smells fishy to me. I suspect there is something going on behind the scenes between Matousec and Comodo.

 

Nothing odd, nothing fishy, your "suspicion" is baseless. I'm fairly confident that you're ignoring Matousec's methodology and rules. To reproduce Matousec's results, you should follow Matousec's methodology and rules - step by step - and you should also test the same product version they tested.

Go study these links:
http://www.matousec.com/projects/proactive-security-challenge-64/#methodology-rules
And especially:
http://www.matousec.com/projects/proactive-security-challenge-64/faq.php

 

I'm aware of their methodology. Difference was I used the latest available version at the time. However, I find it rather impossible for a product to gain ~25% over a single version.
I might retest the latest version when I have the time.

Also, considering Comodo's reputation, past, and even present, I don't think it's at all baseless for me to suspect some foul play.

 

The good thing is, you can DL & run ALL the tests yourself & see how your Apps fare. I've run them ALL in the past, & a Lot more besides & was more than happy with the result, on this older OS & Apps

*

No, but it "appears" to be holding it's own in spite of that Not all older stuff is past it's sell by date

 

Yes, I agree it does very well. It's really an excellent firewall in so many ways. I've a license for the latest version, but I haven't used it for a while, having settled on Windows fw. The lack of development concerns me not because it can't pass all the Matousec tests (I don't really care much about those), but because it indicates the developer may not be focusing on it, which either means he's going to drop it alltogether, or fall behind in keeping it functional with current and future Windows versions, where a monthly patch could, at some point, interfere with the fw's functionality.

 

Remember to read this: http://www.matousec.com/projects/security-software-testing-suite-64/

 

Not much products tested

 

Latest news

• 2012-01-16: Responses from Comodo Security Solutions, Inc., the vendor of Comodo Internet Security Premium, and PWI, Inc., the vendor of Privatefirewall, have been added.

Comodo Security Solutions, Inc. – the vendor of Comodo Internet Security Premium

2012-01-16 (Comodo Internet Security Premium 5.9.219863.2196 scored 94 % and took 1st place): Thank you for including our product in your latest tests. Your open-source, open methodology tests are always very well thought out and unique. The latest results indicate the fact that there is a reason for us to be able to offer a 500 USD virus free guarantee in all operating systems. Keep up the good work!
Egemen TAS,
Director, Desktop Security Products,
COMODO

PWI, Inc. – the vendor of Privatefirewall

2012-01-16 (Privatefirewall 7.0.25.5 scored 56 % and took 3rd place): While a #3 ranking is not very satisfying when combined with the Protection level earned by Privatefirewall in this initial round of the new x64 PSC, we are aware of the work required on our end to dramatically improve our performance and the effort to do so is already underway. We thank the Matousec organization for their continued commitment to helping vendors improve their products and ensuring that the consumers and businesses that rely on them are able to realize the greatest levels of system protection.
Greg Salvato
CEO - Privacyware

Source: http://www.matousec.com/projects/proactive-security-challenge-64/#latest-news

 

Great to see more results and responses.

 

Updated: http://www.matousec.com/projects/proactive-security-challenge-64/

 

Interesting that as poorly as ESET did, they still beat Norton by more than 3 times over. I'd like to see the Russian vendors added to this, they usually do pretty well.

 

This is a HIPS test but still interesting.

 

Oh So Cool NOT

 

Source.