Secunia Test of Internet Security Suites

Is this really a valid test http://secunia.com/gfx/Secunia_Exploit-vs-AV_test-Oct-2008.pdf

 

A bit more info:
http://secunia.com/blog/29/

 

Antivirus solutions won't plug in the holes in your applications no matter how much some people would like to (and then we get into the area of those HTTP scanners and similar wannabe solutions). You'll do yourself a much better service if you install something like Secunia PSI and schedule it to run daily to get alerted about security hotfixes.

(The above completely disregards stuff like vulnerabilities in AV software itself, where the cure proves worse than a disease and in fact those AVs help the attackers compromise affected systems.)

 

While Secunia has a point in claiming that AVs don't focus too much on detecting vulnerabilities. But the detection rate of the AV products would look ALOT better if Secunia had tested exploits that actually download/drop malware and execute it. I am pretty sure most of the HIPS/behaviour blockers are very well optimized for that. Who cares which vulnerability in Word, Excel or PowerPoint allows you to execute code and launch malware - if your HIPS does put Microsoft Office under special surveilance and will catch attempts to drop/launch malware from within the memory of Microsoft Office. That approach is independent of the exploit actually used.

In the end, Secunia wants to sell a product, the message of this test is accordingly.

Besides, just detecting the vulnerablity will somtimes give you false positives on randomly corrupted documents/files. I had quite a few of that with my Office vuln. detections. Try explaining a customer that his precious document is just corrupted and not an actual exploit, even though the corruption exactly triggers the vulnerability...

 

I totally agree with Stefan. They just create some artificial "malware" exploiting vulnerabilities .. but as far as I see this no real damage is done.

If it was real malware programs with excellent detection rates like Avira would probably have cought it.

Furthermore, programs with built in behavior blocker, like KIS, would probably have yelled that something very dangerous is going on / or directly blocked it.

But if there is no real damage .. why block it?

Hence, why this test?

 

well geez, no wonder their tests are crap when they leave out my sig.

 

This test has two types of sub-tests, one where a folder was scanned (which is pretty useless), and a second where they browsed to a web-page that was infected (this is a very valid test).

As expected NIS2009 killed the competition. Notice how poor Kaspersky's detection of exploits is since they don't write generic sigs for the exploits for the most part. Rather to reduce FPs, some parts of the shell-code are included in the signature pattern.. very poor.

This test exposes KIS and all other vendors that dont write generic signatures to detect exploits.

NIS2009 detected 10 times more exploits than Kaspersky and others...K fan boys please note.

 

Though Norton IS apparently trounced the competition, they still failed on 236 exploits out of 300.

With regards to Kaspersky, it should be pointed out KIS 2009 uses the Secunia database when scanning for vulnerabilities.

 

My my, when I get real-world exploit samples from customers (some are targeted attacks), Symantec does not look so shiny (if you can call 20% shiny) anymore - and the rest doesn't fail so "badly" anymore aswell.

What's so bad about detecting shellcode? Of course, it can be easily replaced, but there is a pretty large amount of lazy malware writers that keep using the same shellcode. If you have a good generic shellcode detection, you could even catch a total new exploit if the malware author was too lazy to obscure the shellcode well enough.

 

Why are they using Trend Micro IS 2008 , instead of the 2009 version ? TMIS 2009 was released 3 days after Norton 2009.


Anyways, I agree that scanning for vulnerability exploit code is not a solution. The answer is in ensuring that system is patched from vulnerability or the behavioral/HIPS component can thwart possible exploits.

 

@Stefan K:

What are you referencing there?

 

im a fan of kaspersky and also for avira!!

anyway i will tell the truth kav generic detection is realy weak!! they dont write generic signatures not just for exploits but also for malware!!! thats my big disappointment on kav

even avast has kill it with its own generic detection!!!

 

What are you trying to say ? That Secunia rigged the test. If the rigged the test then it would only be in Kaspersky's favor because KIS2009 uses their technology.

I'll tell you whats so bad.. You have to keep revving the signatures to keep up with all the variants of shell code out there generated by all the polymorphic shell code generators. Kaspersky labs has revved the Psyme signature 1220 tmes http://www.viruslist.com/en/find?search_mode=virus&words=psyme

 

why don't they test ESET SMART SECURITY

 

It is really obvious . Just have a look at the conclusion/final results.

 

I wouldn't be surprised if some of those earlier Psyme detections have now found their way into generic signatures.

 

Jeez and there was me thinking the "poke fun at people because your antivirus has a bigger peni* than their antivirus" was old.

Cant we just discuss the test without the pointless sideswipes all the time

If your favourite product did well in a test be happy but that doesn't give a license to go around proclaiming everything else sucks (which is pretty silly really)


Now...back to your statement

So?

They tested the file scanner. Big deal. Does Kaspersky and the other products tested consist of only a file scanner or multiple layers of security?

It doesn't matter how a file is detected, as long as it stopped. Its the same thing as saying "Hey, that gun wasn't detected by the Xray machine but it was detected by the metal detector at the door"...that means our security is crap(?)

 

Some of you may be interested to read Alex Eckleberry's blog on this test, which include comments from Andreas Marx:

http://sunbeltblog.blogspot.com/2008/10/another-useless-test-grabs-headlines.html

 

http://research.pandasecurity.com/archive/Exploits-vs-Antivirus-_2D00_-The-Last-Stand.aspx

 

Test seem pointless to me. It simple says that Norton has a larger signature database.

I routinely download viruses (I check every malware link I get.) Downloading is not a hazard, more often than not the file downloaded is a variant and quite often the file it self is not detected as malware; however, executing that file triggers alarms, assuming I allow it to run in the first place. Just checking a file is great, but more important is stopping the virus from installing. I would suspect that many of those listed products would have fair much better if these malware files had been executed.

Edit: My Security Suite was not tested; which is fine with me.

 

Secunia exploits security suites flaws :
http://news.cnet.com/8301-1009_3-10066975-83.html

 

AV vendors are crying as most of them persorm poorly in the test.

IMO detection of explits is very important. So I will say it,s sgood test indeed. You will now see the vendors adding sgnatures for the exploits more n more.

 

I totaly agree with you..

And i think most of the user's here says this av test sucks just because their AV did not score good maybe? it's a question?

 

At least this test made the vendors think about this exploits and maybe they will consider detection of them.

 

no, detection of malicious code is very important, not detection of lab-grown exploits... how are you supposed to use those exploits to test how vulnerable your applications are if your av is blocking access to them? the entire premise of the test is ridiculous on that basis alone...

the fact that they misused the security suites means their methodology was retarded, and the fact that secunia has a financial incentive to try and tarnish the reputation of av vendors just makes it all the more like some unethical marketing exercise...