New Detection Test - Dennis Labs

[dschrader]

We at Symantec engaged Dennis Labs to do a new type of test of security effectiveness. The results can be found here:

http://community.norton.com/norton/a...t-consumer.pdf

I know . . . I know . . . testing paid for by a vendor is suspect.

But the results are worth at least looking at.

We are trying to address the problem that the major labs - av-comparatives, av-test, VB, ICSA, West Coast - none of them test what we consider "real-world" scenarios. Most of these tests are of zoos of malware sitting on hard disk. This simply isn't how most users encounter viruses. So we asked Dennis Labs to identify malware infected sites and to surf those sites with 10 different internet security suites installed - and to record the full experience.

The results are interesting in that where most products score at near 100% detection on the zoo tests - more then half scored 75% or below on the Dennis Labs results.

I'm not knocking av-comparatives and av-test - those tests are valuable. But they don't tell the whole story. We need independent labs doing Dennis Labs type tests. It is time consuming and expensive - but it will fill in an important gap in comparing vendor claims.

Dan
Symantec

[Fajo]

Those test seem well. Screwed up! lol. And quite frankly it coming directly from Symantec Seems like the tests were to make sure symantec detected it first. In other words if norton missed a site or a Trojan it was simply removed from the test. We all know no AV is perfect and will NOT score 100% on a test unless its rigged or very very one sided.

[trjam]

Why tests like this are BS as the post, if Norton had been ranked where you say Panda is, would you have been so kind to let us know this. Hell no.

Also, where are certain other AV products that are fairly good at zero protection, say Eset? Well we did not include them because then we would have not been first. I am 54 years old and have finally learned one thing in life, just one thing.

Norton, Symantec, whatever you want to call them are so full of sh*t they will never, ever get my money. What was this suppose to accomplish. If you want to test zero day malware use the top 20 AVs, use a few HIPS products and some behavorial blockers and lets really see where Norton stands.

this is why we do trust av-comparatives, av-test, VB, ICSA, West Coast instead of your crappy software.

[chris2busy]

Why oh why,would they test the personal version of avira,which is deprived of spyware detection?
(avast and avg free editions detect fully whatever their full version do as well ,just lack some bells n whistles)

[trjam]

Lol, not including a major player in the market like Eset, tells me where they finished. Will you folks ever learn. Kudos to Eset in zero day protection. If you adjust your dislay settings from 800 by 600 pixels, to 1280 by 1024, you will actually see they are just a little left of Norton.

[pbust]

Without getting into the validity of sponsored tests where the sponsor gets to define methodology and pick & choose testbed samples, I noticed this in your methodogy description:

In order to improve the chances that each target system received the same experience from a malicious web server, we used a caching proxy set to ‘offline’ mode and a web replay system. When the first target system visited a site, the page’s content, including malicious code, was downloaded and stored. When each consecutive target system visited the site, it should have received the same content, with some provisos.

Did you test what this type of setup (caching proxy in offline mode) does to HTTP-based cloud-scanning technologies such as the one implemented in our product? I mention this as there are quite a few known problems that could affect HTTP-based cloud-scanning performance in caching proxies. See RFC 3143 for some details.

[trjam]

Originally Posted by pbust

Without getting into the validity of sponsored tests where the sponsor gets to define methodology and pick & choose testbed samples, I noticed this in your methodogy description:

Did you test what this type of setup (caching proxy in offline mode) does to HTTP-based cloud-scanning technologies such as the one implemented in our product? I mention this as there are quite a few known problems that could affect HTTP-based cloud-scanning performance in caching proxies. See RFC 3143 for some details.

no PB, why would they ask a paid testing firm to do that. Geez, you want Panda to actually look good or something.

This is Norton, and they stand the most to lose to Microsoft and their new approach to security, which is about as crappy as Norton. VBA? You folks have nothing to worry about in the future, trust me. Big, equates to ignorance.

[Pleonasm]

Dan (“dschrader”), I congratulate you and Symantec for efforts to add realism to the testing of anti-virus products. From my perspective, the methodology used in the test is a close approximation to assessing “real world” protection -- although this test (like all others) has limitations.

Originally Posted by Fajo

In other words if Norton missed a site or a Trojan it was simply removed from the test.

Is there any evidence that the accusation is true?

Originally Posted by trjam

Also, where are certain other AV products that are fairly good at zero protection, say Eset?

Good question. What were the criteria employed in the selection of anti-virus products tested?

[ratchet]

Originally Posted by trjam

Lol, not including a major player in the market like Eset, tells me where they finished. Will you folks ever learn. Kudos to Eset in zero day protection. If you adjust your dislay settings from 800 by 600 pixels, to 1280 by 1024, you will actually see they are just a little left of Norton.

Actually, I was going to make the point that NOD is conspicuous by its absence! I've seen a test where definitions are held back several weeks and then they hit the antis with "wild" malware and no one ever comes close to NOD in that test.

[Pleonasm]

Originally Posted by trjam

Geez, you want Panda to actually look good or something

Panda, if it so chooses, has the option of repeating the same well documented methodology (or improving that methodology) with an independent testing organization of its own choice and reporting the results. If Panda believes that these findings are inaccurate or misleading, then I encourage Panda to do so.

The general thrust of the test reported by Symantec, while not perfect, seems to be a reasonable approximation of "real world" activity that an actual user might encounter.

[lodore]

I cant seem to find out what sort of settings each av has been set to?

[funkydude]

Originally Posted by lodore

I cant seem to find out what sort of settings each av has been set to?

edit: removed since lodore edited hes post 5 million times.

But seriously, only 40 samples, beta software, no live internet connection. Sooooo many flaws, so unrealistic.

[pbust]

Report mentions the full report is available in Excel with notes. However I can't find it on Dennis website (http://www.dennis.co.uk) nor Symantec's. Anybody know where to get it?

There are many test cases (pgs. 19-33) where there is an alert and action (blocked, neutralized, denied access or deleted) but yet the report counts them as "compromised":

7 PIS Toaster Blocked Multiple (see notes) Report Quarantined Suspicious file
8 PIS Toaster Neutralized Multiple - see notes Report Deleted Multiple
9 PIS Toaster Blocked Dangerous operation blocked! Report Deleted Adware
12 AVA Pop-up Abort connection S:Obfuscated-DQ (Trj) Report Move to Chest Multiple
14 KIS Pop-up Suspicious activity Suspicious driver installation Report Quarantined Multiple
15 PIS Pop-up Delete Exploit/DirektShow.A Report Multiple - see notes Multiple - see notes
20 BDF Pop-up Blocked Trojan.SWF.Dropper.C Report Multiple - see notes Multiple - see notes
20 PIS Toaster Blocked Dangerous operation blocked! Report Multiple - see notes Multiple - see notes
22 PIS Toaster Multiple - see notes Multiple - see notes Report Quarantined Suspicious file x3
23 AVI Pop-up Deny access Multiple - see notes Report Found Hidden objects x2
24 PIS Toaster Blocked Dangerous operation blocked! Report Deleted systemguard2009
33 PIS Pop-up Deleted Exploit/DirektShow.A Report Multiple - see notes Multiple - see notes
40 AVI Pop-up Deny access Multiple - see notes Report Repair all Multiple - see note
40 MIS Pop-up Blocked Buffer overflow Report Quarantined Artemis!59EBBE31B3AF

I'd like to understand how a deleted or quarantined threat is treated as an actual compromise.

[pbust]

Even weirder, there are some test cases for NIS (Norton) where it didn't alert nor block the threat, yet it is counted as "complete remediation" and "defended".

1 NIS None None None None None None
5 NIS None None None None None None
8 NIS None None None None None None
9 NIS None See note None None None None
13 NIS None None None None None None
21 NIS None See note None None None None
22 NIS None See note None None None None
23 NIS None See note None None None None
24 NIS None See note None Report Removed 2 tracking cookies
29 NIS None See note None Report Removed 2 tracking cookies
33 NIS None None None n/a n/a n/a
39 NIS None None None n/a n/a n/a


Unless I'm reading this wrong, according to the actual results shown on the table on pages 19-33 Norton was awarded "complete protection" on 12 test cases where there was no detection whatsoever and which should probably read "compromised".

Can someone else please look at this to make sure I'm reading it correctly?
dschrader, are you there?

[dschrader]

To those that want ESET, NOD, Dr Ah, Malwarebytes . . . . I would love to have a test with a comprehensive set of security products. Actually, I would love to have the budget to do that test.

The fact is that this type of testing is expensive and time consuming. We had to make some hard choices. Our team in Japan lobbied for Sourcenext - who dominates that market, our China group wanted Rising, Eastern Europe wanted ESET, Europe wanted G-Data . . . . we have something in the range of 26 competitors that could have been included. We choose the products that felt most important to us either due to installed base or because of perception of technology that we wanted to test.

Ideally an industry group or an independent outfit would do this - it is a big job. Anyone here willing to pitch in to pay for it?

As for those that say the test is useless because Symantec paid for it . . . where did you get to be so cynical?

The results are valid and repeatable. pbust, yes the cloud scanning technologies worked just fine in this setup - both ours and those of Panda, McAfee . . . . To quote the report, "An HTTP replay system ensured that all target systems received the same malware as each other. It was configured
to allow access to the internet so that products could download updates and communicate with any available ‘in the cloud’ servers."

I'm not claiming that this is the last word in testing - but we need to get past the idea that putting a bunch of malware in a directory and scanning it gives meaningful results.

[andyman35]

Leaving aside the remarkably low score for Avira that's completely out of sync with just about every other comparitive test performed in the last 2 or 3 years,can somebody please explain the reasoning behind comparing full suites like NIS against standalone AVs such as Avira and Avast.

[pbust]

Originally Posted by dschrader

pbust, yes the cloud scanning technologies worked just fine in this setup - both ours and those of Panda, McAfee . . . . To quote the report, "An HTTP replay system ensured that all target systems received the same malware as each other. It was configured to allow access to the internet so that products could download updates and communicate with any available ‘in the cloud’ servers."

Those technologies vary a lot. McAfee's for example is based on DNS queries, while ours is based on HTTP. Obviously a caching HTTP proxy in offline mode can have an effect on HTTP cloud-scanning while at the same time have no effect on DNS cloud-scanning. Can you tell me what caching proxy and config you used so we can replicate if it affects *our* cloud scanning?

Originally Posted by dschrader

I'm not claiming that this is the last word in testing - but we need to get past the idea that putting a bunch of malware in a directory and scanning it gives meaningful results.

I couldn't agree more with you, really. But can you please comment on posts #14 and #15 and clarify how come a "miss" on NIS counts as "defended" while a "detection" on other engines counts as "compromise"? What am I missing here?

[ronjor]

I was under the impression Symantec was a member of AMTSO and as such was working hand in hand with other vendors to to improve testing procedures.

Is your pdf file a result of of such an alliance?

[Page42]

Originally Posted by dschrader

As for those that say the test is useless because Symantec paid for it . . . where did you get to be so cynical?

You answered your own question. We learned our cynicism from observing companies like Symantec.

[Zimzi]

Originally Posted by lodore

I cant seem to find out what sort of settings each av has been set to?

In the three best participants were the maximum protection settings while in others real time protection was off?

[the Tester]

AV vendor "engaging" labs to do testing= an advertisement.
This type of "testing" can't be taken seriously imo.

[Fajo]

Originally Posted by Page42

You answered your own question. We learned our cynicism from observing companies like Symantec.

For once me and Page42....... Agreed!

[subset]

Who or what is Dennis Labs

Simon Edwards writes like he is involved in Dennis Virus Labs.
http://simonedwards.blogspot.com/200...virus-lab.html

He performs in a promotional film for Symantec and performs weird promotional "tests" for Symantec.
Oh well, he is also a member of AMTSO.

Brave new malware testing world.
Every vendor pays his own no name testing lab to make his product shine.
Well, that's okay, maybe a smug self-satisfaction.
But not of further interest for the public.

Cheers

[Fajo]

Originally Posted by subset

Who or what is Dennis Labs

Simon Edwards writes like he is involved in Dennis Virus Labs.
http://simonedwards.blogspot.com/200...virus-lab.html

He performs in a promotional film for Symantec and performs weird promotional "tests" for Symantec.
Oh well, he is also a member of AMTSO.

Brave new malware testing world.
Every vendor pays his own no name testing lab to make his product shine.
Well, that's okay, maybe a smug self-satisfaction.
But not of further interest for the public.

Cheers

Its of interest to them simply because now they can slap another sticker on a box and people that don't know any better will go "That one has more awards lets buy it."

GG Symantec Advertising but don't bring it here.