Proactive Security Challenge

http://www.matousec.com/projects/proactive-security-challenge/results.php

Products tested against the suite with 148 tests

even comodo are on this test

 

Hey, Dr Web beat Eset,Avira and Avast. Cool

 

it is enough to read, under Methodology and rules/Installation and configuration, how much room they have left themselves to interpret, distort and massage the results.

 

Plenty of discussion in here on assessing how distorted and incomplete that assessment is...
They are now trying desperately to have a credibility 'wash' but with little success. IMO, a site and approach destined to die in the near future.

 

BIS2011 got such a great score!!! I am surprised...

 

Reacting to intrusion vectors, is in my mind very reactive, not pro-active at all.

The normal sequence of events of malware

0. Deliver an intrusion on a website or through mail (or portable data carrier like USB or CD/DVD etc). Easiest way are websites, since the website contains code (javascript, XML, meta data in pictures/videos, etc) which is executed when visiting this site.
1. Bufferoverflow/stack or memory intrusion = get control (when used with exploit also get elevated rights).
2. Download malware code
3. Execute (or when social engineering is involved, the user him/herself takes care of elevating the malware code to get full control)
4. Install to survive re-boot

Matousec tests step three to four. Not the earliest moment of defense I would say. Even LUA + SRP or LUA + applocker would prevent this. The OS (even Home Premium versions) offer sufficient protection through features or commands of the OS itself.

Safe-Admin for instance takes the approach to:

1) Get control
Your mail and browser run in protected mode (even FF and Opera will get these goodies). Low rights/protected mode processes can't touch other (Medium = User and High = Admin + System) processes. So these intrusions can only harm the browser or the E-mail program itself.
Since 32 bits internet facing software is also virtualised (using RunAsInvoker trick) by Windows (Vista/7) own policy sandbox. These changes to Windows and Program Files as of the Registry are contained with Windows own build in virtualisation. So only the exploited internet facing program would suffer from the intrusion, because other programs still use the original data/ and registry keys. So Low Rights and Windows own Virtualisation really limit the impact of an intrusion.
With Microsoft's own EMET on top of this (also providing DEP, SEHOP, Address Space Randomisation) = the attack surface of these types of exploit is reduced substantionally. In a worst case scenario (very unlikely to happen, I have no intrusion after three months of malware links hunting) the payload would be planted and malware code is downloaded to your harddisk.

2) Download
1806 trick does prevent executables from downloading by IE and FF. So with these browsers, the intrusion script ends here. When code does not download it can not execute. So a far more effective approach than an old school HIPS.

3) Execute
Deny execute of mail and download directories will prevent downloaded malware to execute without user intervention. In the unlikely event that 1 and 2 failed, it all ends in the deny execute cage of download and mail directory. When using Chrome, moving it out of the directory, still has the 1806 execution block for Explorer. When malware code can't execute it can't install.

4) Install
Safe-admin hardens UAC protection by:
- only allowing signed drivers to install
- only allowing signed programs to elevate
- auto elevate only from safe places (Windows/Program Files)
- disable intelligent installer detection (nowadays every software should have a manifest telling what rights it needs, so this Vista migration solution for old XP programs is now often misused)

Bottem line

When I run Matousec tests with safe admin (and allow to download and execute, so I have voluntary opened two entry gates of my defense), I fail on three tests. When I would simulate Safe-admin's low rights protection by assiging the Matousec tests low rights before execution and apply FW application filtering for outbound also: I only fail on ONE DNS test.

So when I have NO security software I am still able to reach level 10+ and get 99%. protection by using a few of the OS mechanismes. This clearly illustrates how useless Matousec tests are. It can't differentiate the added protection of third party software from using the OS-internals.

Other example is that they run Comodo on default setting. For maximum protocol/Firewall protection, you have to enable a few settings in teh firewall. Mind you: I am telling Commodo is ABLE to provide real strong Firewall filtering protection, ONLY this settings are not maxed out by default. Comodo reaching 100% with its Firewall filtering capability on 50% of its protection power is the second clue: Matousec tests are useless!

It is propblably also the reason they changed the test from firewall to pro-active, since 'pro-active' is a test class category of Antivirusses (meaning zero day exploit is not commenly known in blacklist data bases of Anti-virus vendors, so protection is based on 'other' or proactive blacklist defense additions as generic malware family footprint, heuristics or behavioral monitoring). Only Matousec tests POC's not 0-day real threats. So they should call it "Matousec old fashioned lazy POC test to evaluate how third party protects on XP and older Operating Systems from possible malware intrusion vectors". For me the title does not cover the body content it announces, or the 'flag' does not cover the 'cargo' it delivers. In the connected - internet world, the worst fault is to deliver something different you promise. Or in simple terms readers deceit. For me a reason to blacklist them from informative content discussed in this forum.

Request to Wilders himself and the admins

Can we have a new section please with useless tests? On my rating Matousec would be on place 2, just after the test ratings on just the number of exploits with no analysis of their impact.

The now dead German forum Schein-sicherheit (fake security) informed its members on the marketing/advertising features which were of no of very limited use. Wilders is also a security forum. It has a responsibility in informing and educating the general public. I therefore ask the Admin's: please take your responsibility and dare to classify some tests as bullocks.


Thanks Kees

 

It's okay as a HIPS test, but it doesn't tell anything about protection as a whole. Most of those tested products don't have HIPS so of course their scores are low (while they still protect well, just look at all dynamic tests).

Oh, they're trying something? By removing Online Armor and by trying to get money from vendors with their KHOBE article? I don't think they're even trying to fix their credibility.

 

There's nothing more to say ...

 

The problem is that in the absence of many regular,decent tests of this type then Matousec continues to be publicized and discussed,whereas it should really be confined to the dustbin of history .

 

Completely agree, their tests are bogus.
Any site that hides it registrant behind proxy servers in order to disguise its true identity is not trusted. It has been said but not proven that Comodo actually owns matousec.com, who knows? it could be true since matousec.com uses Domains by Proxy, Inc.- http://whois.domaintools.com/matousec.com

Maybe it was cheaper for Comodo to buy matousec.com then to keep paying for re-testing.

Thanks.

 

LOL... but a possibility...

 

They are still testing cis 4
accordng to this test whoever not using comodo is a moron
good to see funny tests like these

 

According to your argument with every single test if you are not using the product ranked nº1 you are a moron... ergo everybody is a moron.

 

Who decides number 1 ? each test has a different no.1 ... finally its upto the user ... a will tell x is number 1 b will tell y is number 1...everyone has there own no 1 product so no one is moron

 

Well this is a very different argument.

 

i was not actually bashing comodo..i was bashing the test...i hope u got my point now

 

maybe people at comodo support forum see things in a different way... LOL
probably what melih was after: a test were he can achieve 100%