Secunia Test of Internet Security Suites

This is a follow-up on the Secunia tests by Aleks at Kaspersky:

October 17, 2008 | 12:57 GMT

By now most people have seen the Secunia test results and all the ensuing discussions. Frankly, I was a bit surprised by the vehemently negative reaction from a number of AV vendors.

And it doesn't seem to be about the 20% difference between the 'winner' and the rest. Criticism has focused on the testing methodology, which many people thought was dubious. Some of the suggestions were useful - mostly those from Andreas Marx, the well-known AV solutions tester from Germany. The general tone, though, seems to be that many AV vendors thought their results would have been a lot better if the test methodology had been different. And maybe they're right.

But I think people are too focused on looking for mistakes in the tests and/or attempting to explain their poor PoC detection rates. Sure, criticizing Secunia's testing methods is justified, but only if we're discussing testing methodology, and nothing else.

As I see it, Secunia wasn't trying to highlight the weaknesses of AV solutions - I think they were trying to make a different point...

At Kaspersky, we've taken a decision not to detect PoC vulnerabilities - it's far more sensible to focus on protecting users from the real threats and exploits that are being used by malware authors in the real world. That's what our antivirus databases are for. The point isn't so much that detecting PoCs is a pretty difficult task (although the test results clearly show that even Microsoft and Symantec, with all of their resources, didn't fare all that well) but that detecting PoC s is a dead end, and doesn't address the fundamental problem.

So what is the problem?

An abundance of vulnerable applications. And the solution for this problem doesn't lie in detecting 65% or even 99% of PoCs. Nor does it lie in good or bad AV testing methodology. The only real solution is proper patch management. In the context of the post test discussion, I get the feeling that a lot of people are conveniently forgetting or ignoring Secunia's "What to do" list:

Users and businesses need to take the threat seriously and realise that firewalls and traditional security software, such as that included in Internet Security Suites, isn't sufficient to protect PCs and corporate networks.
Because the security industry can never offer a protection that matches that of a properly patched program, consumers and businesses have to put more effort into patching their programs. If your programs are vulnerable and unpatched, then you're left quite exposed to new attacks.
What makes patching even more attractive is the fact that it is free-of-charge. It only costs the amount of time invested in downloading and installing the patch/update. With tools such as the free Secunia Personal Software Inspector (PSI) and the similar functionality offered by Kaspersky Internet Security 2009 it is very easy to identify the programs that needs patching.

Fortunately, the AV industry is taking steps to tackle the patching issue. Our product, Kaspersky Internet Security 2009 is so far the first and only product to contain a vulnerability scanner. It identifies applications that have unpatched vulnerabilities - a log gives details of the vulnerability, including a name, threat level and what needs to be done to install the necessary patches.

This is just a first step towards a fully-functional system for managing risk on personal computers, and we're going to continue active work in this area.
We need to treat the disease, not the symptoms. In this case, the disease is all the vulnerable applications which pose a potential risk that is exacerbated by users' lack of knowledge. And this is not something the AV can, or should, tackle alone - it's a matter of security in general.

Moreover, no AV vendor, no matter how well they do on such tests, has the right to say 'Great, we protect you against all exploits, so you needn't patch'. No company would dare say this, and everyone agrees patching is necessary. This fact alone leaves those who are hotly discussing Secunia's test results and methodology without a leg to stand on.

We're happy about the increased awareness of vulnerabilities and the responsibilities of AV vendors that we're seeing. The AV industry can't begin solving the problem of patching soon enough for me. We need both new technologies and user education - we need to talk about patch management until home users understand that it plays just as big a role in security as AV software does.

Patch management begins with the head, and not with the software.

http://www.viruslist.com/en/weblog

 

I find it surprising that they missed Eeye Blink. Which advertises to be a suite which provides protection against vulnerabilities.

 

looks like he still doesn't get that if you're testing suites you should test all parts of those suites.... otherwise you aren't really measuring the protective capability of the entire suite...

 

First at all, thank you for that kind word (BS). I can see you've misunderstood what I've written, and that's because most likely I haven't written it rightly (english is not my native language).

I'll try to explain better my thoughts.

First: This test that Secunia has done is quite interesting, at least in my humble opinion. The ability for a security suite to prevent exploit attacks is quickly becoming a priority. I personally think an exploit is always intended to be "malicious", both if it's only a PoC or it has a malicious payload. An exploit is, by definition, a piece of software, a chunk of data, or sequence of commands that take advantage of a bug, glitch or vulnerability in order to cause unintended or unanticipated behavior to occur on computer software, hardware, or something electronic. Unintended or unanticipated, that would mean something unwanted and not explicitly asked by the user. In other words, something that should be detected.

Now, the problem arises on how to detect exploit attacks. As I've already told, that's not trivial detecting exploits basing the detection only on a plain signature. I've no doubt Symantec can detect RDS.Dataspace or every other HTML/JS/VBS exploit (that wouldn't be totally true, if you agree with Secunia test).

Though when you come to other kind of exploits, then other problems appear. First, as Stefan said in one of his posts, the problem of FPs. Just an example: corrupted files that "look like" but, instead, they are only corrupted files. You've to explain then to your users all these FPs, and sometimes you've to apologize for them.

Often is choosed to add signatures only for those exploits that are really in the wild and used to spread malicious softwares, leaving all the rest to other proactive technologies. Here you can read what Panda Software (and Mike on an earlier post) was saying.

When an exploit is executed, then you could detect it in a easier and more generic way than adding specific signatures for every new exploit discovered. Term "executed" doesn't necessarily mean that it has already run its malicious payload (if there's one) and it has infected the machine.

That's what Pedro Bustamante was saying on the Panda Software blog, that their software use generic heuristic rules to proactively prevent and block exploits...but the exploit has to be run! If it isn't, then it's only a piece of code that can be detected only using plain signatures (With all their pro and cons).

Moreover, when you test security suites, you've to test the whole suite. If you test only the antivirus engine, than it's not a test of the suite. Security suites are designed to give the user more layers of security, to try to prevent every kind of attack. What does it mean? For instance that an exploit that could have bypassed the antivirus engine is then blocked by (just an example) an HIPS module. Or, for instance, the malicious payload is then heuristically blocked. And so on. That's how security suites are designed, to give more layers of security to the user.

In the end, what I want to say with this post is that the test done by Secunia is in my opinion a good idea but it has been run in the wrong manner. You can say: "They have run html based exploits". Yes, true. That's what they should do for every kind of exploit, to give a full overview of security suite's abilities.

I hope I've made my thoughts expressed in my previous post a bit more clear.

Best regards

 

It's clearer than water.
Thanks for posting the article

 

I agree. Half the test is wrong, but the other half where they tested web based exploits is correct. So even if Panda could detect exploits when they are run, then they should have been able to detect the web-based exploits because they were in fact run. But they didn't, so I think Panda is trying to blow smoke up everybody's a*s.

There were 157 web-based exploits that were all "run". Panda detected ZERO!! Kaspersky detected ZERO!

 

Yep. Thats the Green K's PR machine in over-drive. Its one think building a product that tells you that your patches are missing (heck.. Microsoft even does that for their products). Its another thing to find all those patches from dozens of different websites and automatically run them (which Kaspersky doesn't do).

And.. its yet another for the customers to even know that they need to run the vulnerability scanner. Overall IMO Kaspersky's approach is a PR gimmick.

Symantec has the best vulnerability database in the world with SecurityFocus. They could easily build such a vulnerability tool. Why do you think they aren't building one ? Because it DOESN't WORK!

 

Oh btw.....as for the thread title..

Secunia Test: gently brought to you by S_ _ _ _ _ _ c .

 

Actually its a cleverly orchestrated marketing stunt by Secunia and K_ _ _ _ _ _ _ y. Just look at K's press release after the "test". Sure does help "K" prove their point that there is a HUGE need for vulnerability assessment (which does not work in practice btw), that COINCIDENTALLY, Secunia happens to be selling.

How convenient

 

I really think that a majority of the posts on this topic are a trifle harsh. Secunia's point for a while has been that computer users are ignoring a facet of security that is very important and very easily fixed, and that is keeping up to date versions of code that malware writers love to hack into (ie Quicktime, Flash, etc).

I've been using Secunia PSI for quite a while (free, by the way) and it has always alerted me to updates that were critical and would only be found by reading security news.

PSI is a very valuable tool for any out there who don't know what versions of Flash, etc they are currently using and who aren't sure what the most updated version is. To bash a free tool that will provide this information seems to me to be counterproductive especially on a Security forum.

 

it's a shame this ignores how a great many people in security in enterprises work... they use exploits in order to test the vulnerable surface area of their systems so as to better prioritize patches (because they often have way more patches than they can easily deal with) and to make sure patches actually work... that won't be possible if the anti-malware industry starts adding detection for benign exploits...

ignoring these customers will hurt the industry and ultimately hamper security...

 

Well said. Patch management should be an integral part of maintaining security on your system. That is the point of secuina's test. Why are we bagging out certain AVs?

 

I ran the PSI tool for a test drive, and it seems I'm "patched". The KL vulnerability scanner doesn't alert me to missing patches either. I suppose this is because I've kept my programs current with the latest releases. That's not to say a vulnerability may appear between now and the next release, but I can't do anything about that till the patch/software update is available.

By keeping on top of things, you're narrowing the chances of getting hit, and I think this is the whole point with regards to patch management.

It's worth noting that by the time vulnerabilities are disclosed by the likes of Secunia, patches or updated versions of the software are usually available.

 

PSI only alerts you if the patches exist. The whole point of it is to alert you to update, not to expose vulnerabilities.

It's useful, not theoretical.

 

Not exactly; it alerts you of end-of-life and unpatched vulnerable stuff as well as long as it's in their vulnerabilities DB. And yeah, it's useful of course, don't really get all the fuss here...

 

Secunia obviously knows how to choose their technology partners

Go Kaspersky go!

 

Yep; they sure did pick the "winning" horse

 

Well kudos to Symantec. NIS09 obviously has definitions for exploits; Symantec has a whole library of "bloodhound.exploit.XX" definitions, and I actually stumbled in one of those detections recently.