New Detection Test - Dennis Labs

Discussion in 'other anti-virus software' started by dschrader, Oct 29, 2009.

Thread Status:
Not open for further replies.
  1. dschrader

    dschrader AV Expert

    Joined:
    Mar 10, 2009
    Posts:
    54
    We at Symantec engaged Dennis Labs to do a new type of test of security effectiveness. The results can be found here:

    http://community.norton.com/norton/...Virus-Protection-2010-DTL-Report-consumer.pdf

    I know . . . I know . . . testing paid for by a vendor is suspect.

    But the results are worth at least looking at.

    We are trying to address the problem that the major labs - av-comparatives, av-test, VB, ICSA, West Coast - none of them test what we consider "real-world" scenarios. Most of these tests are of zoos of malware sitting on hard disk. This simply isn't how most users encounter viruses. So we asked Dennis Labs to identify malware infected sites and to surf those sites with 10 different internet security suites installed - and to record the full experience.

    The results are interesting in that where most products score at near 100% detection on the zoo tests - more then half scored 75% or below on the Dennis Labs results.

    I'm not knocking av-comparatives and av-test - those tests are valuable. But they don't tell the whole story. We need independent labs doing Dennis Labs type tests. It is time consuming and expensive - but it will fill in an important gap in comparing vendor claims.

    Dan
    Symantec
     
  2. Fajo

    Fajo Registered Member

    Joined:
    Jun 13, 2008
    Posts:
    1,814
    Those test seem well. Screwed up! lol. :blink: And quite frankly it coming directly from Symantec Seems like the tests were to make sure symantec detected it first. In other words if norton missed a site or a Trojan it was simply removed from the test. We all know no AV is perfect and will NOT score 100% on a test unless its rigged :cautious: or very very one sided.
     
    Last edited by a moderator: Oct 29, 2009
  3. trjam

    trjam Registered Member

    Joined:
    Aug 18, 2006
    Posts:
    9,102
    Location:
    North Carolina USA
    Why tests like this are BS as the post, if Norton had been ranked where you say Panda is, would you have been so kind to let us know this. Hell no.:cautious:

    Also, where are certain other AV products that are fairly good at zero protection, say Eset? Well we did not include them because then we would have not been first. I am 54 years old and have finally learned one thing in life, just one thing.

    Norton, Symantec, whatever you want to call them are so full of sh*t they will never, ever get my money. What was this suppose to accomplish. If you want to test zero day malware use the top 20 AVs, use a few HIPS products and some behavorial blockers and lets really see where Norton stands.

    this is why we do trust av-comparatives, av-test, VB, ICSA, West Coast instead of your crappy software.
     
  4. chris2busy

    chris2busy Registered Member

    Joined:
    Jun 14, 2007
    Posts:
    477
    Why oh why,would they test the personal version of avira,which is deprived of spyware detection?
    (avast and avg free editions detect fully whatever their full version do as well ,just lack some bells n whistles)
     
  5. trjam

    trjam Registered Member

    Joined:
    Aug 18, 2006
    Posts:
    9,102
    Location:
    North Carolina USA
    Lol, not including a major player in the market like Eset, tells me where they finished. Will you folks ever learn. Kudos to Eset in zero day protection. If you adjust your dislay settings from 800 by 600 pixels, to 1280 by 1024, you will actually see they are just a little left of Norton.:cautious:
     
  6. pbust

    pbust AV Expert

    Joined:
    Apr 29, 2009
    Posts:
    1,176
    Location:
    Spain
    Without getting into the validity of sponsored tests where the sponsor gets to define methodology and pick & choose testbed samples, I noticed this in your methodogy description:
    Did you test what this type of setup (caching proxy in offline mode) does to HTTP-based cloud-scanning technologies such as the one implemented in our product? I mention this as there are quite a few known problems that could affect HTTP-based cloud-scanning performance in caching proxies. See RFC 3143 for some details.
     
  7. trjam

    trjam Registered Member

    Joined:
    Aug 18, 2006
    Posts:
    9,102
    Location:
    North Carolina USA
    no PB, why would they ask a paid testing firm to do that. Geez, you want Panda to actually look good or something.;)

    This is Norton, and they stand the most to lose to Microsoft and their new approach to security, which is about as crappy as Norton. VBA? You folks have nothing to worry about in the future, trust me. Big, equates to ignorance.
     
  8. Pleonasm

    Pleonasm Registered Member

    Joined:
    Apr 9, 2007
    Posts:
    1,201
    Dan (“dschrader”), I congratulate you and Symantec for efforts to add realism to the testing of anti-virus products. From my perspective, the methodology used in the test is a close approximation to assessing “real world” protection -- although this test (like all others) has limitations.

    Is there any evidence that the accusation is true?

    Good question. What were the criteria employed in the selection of anti-virus products tested?
     
  9. ratchet

    ratchet Registered Member

    Joined:
    Feb 20, 2006
    Posts:
    1,988
    Actually, I was going to make the point that NOD is conspicuous by its absence! I've seen a test where definitions are held back several weeks and then they hit the antis with "wild" malware and no one ever comes close to NOD in that test.
     
  10. Pleonasm

    Pleonasm Registered Member

    Joined:
    Apr 9, 2007
    Posts:
    1,201
    Panda, if it so chooses, has the option of repeating the same well documented methodology (or improving that methodology) with an independent testing organization of its own choice and reporting the results. If Panda believes that these findings are inaccurate or misleading, then I encourage Panda to do so.

    The general thrust of the test reported by Symantec, while not perfect, seems to be a reasonable approximation of "real world" activity that an actual user might encounter.
     
  11. lodore

    lodore Registered Member

    Joined:
    Jun 22, 2006
    Posts:
    9,065
    I cant seem to find out what sort of settings each av has been set to?
     
    Last edited: Oct 29, 2009
  12. elapsed

    elapsed Registered Member

    Joined:
    Apr 5, 2004
    Posts:
    7,076
    edit: removed since lodore edited hes post 5 million times.

    But seriously, only 40 samples, beta software, no live internet connection. Sooooo many flaws, so unrealistic.
     
    Last edited: Oct 29, 2009
  13. pbust

    pbust AV Expert

    Joined:
    Apr 29, 2009
    Posts:
    1,176
    Location:
    Spain
    Report mentions the full report is available in Excel with notes. However I can't find it on Dennis website (http://www.dennis.co.uk) nor Symantec's. Anybody know where to get it?

    There are many test cases (pgs. 19-33) where there is an alert and action (blocked, neutralized, denied access or deleted) but yet the report counts them as "compromised":

    Code:
    7 PIS Toaster Blocked Multiple (see notes) Report Quarantined Suspicious file
    8 PIS Toaster Neutralized Multiple - see notes Report Deleted Multiple
    9 PIS Toaster Blocked Dangerous operation blocked! Report Deleted Adware
    12 AVA Pop-up Abort connection S:Obfuscated-DQ (Trj) Report Move to Chest Multiple
    14 KIS Pop-up Suspicious activity Suspicious driver installation Report Quarantined Multiple
    15 PIS Pop-up Delete Exploit/DirektShow.A Report Multiple - see notes Multiple - see notes
    20 BDF Pop-up Blocked Trojan.SWF.Dropper.C Report Multiple - see notes Multiple - see notes
    20 PIS Toaster Blocked Dangerous operation blocked! Report Multiple - see notes Multiple - see notes
    22 PIS Toaster Multiple - see notes Multiple - see notes Report Quarantined Suspicious file x3
    23 AVI Pop-up Deny access Multiple - see notes Report Found Hidden objects x2
    24 PIS Toaster Blocked Dangerous operation blocked! Report Deleted systemguard2009
    33 PIS Pop-up Deleted Exploit/DirektShow.A Report Multiple - see notes Multiple - see notes
    40 AVI Pop-up Deny access Multiple - see notes Report Repair all Multiple - see note
    40 MIS Pop-up Blocked Buffer overflow Report Quarantined Artemis!59EBBE31B3AF
    I'd like to understand how a deleted or quarantined threat is treated as an actual compromise.
     
    Last edited: Oct 29, 2009
  14. pbust

    pbust AV Expert

    Joined:
    Apr 29, 2009
    Posts:
    1,176
    Location:
    Spain
    Even weirder, there are some test cases for NIS (Norton) where it didn't alert nor block the threat, yet it is counted as "complete remediation" and "defended".

    1 NIS None None None None None None
    5 NIS None None None None None None
    8 NIS None None None None None None
    9 NIS None See note None None None None
    13 NIS None None None None None None
    21 NIS None See note None None None None
    22 NIS None See note None None None None
    23 NIS None See note None None None None
    24 NIS None See note None Report Removed 2 tracking cookies
    29 NIS None See note None Report Removed 2 tracking cookies
    33 NIS None None None n/a n/a n/a
    39 NIS None None None n/a n/a n/a


    Unless I'm reading this wrong, according to the actual results shown on the table on pages 19-33 Norton was awarded "complete protection" on 12 test cases where there was no detection whatsoever and which should probably read "compromised".

    Can someone else please look at this to make sure I'm reading it correctly?
    dschrader, are you there?
     
  15. dschrader

    dschrader AV Expert

    Joined:
    Mar 10, 2009
    Posts:
    54
    To those that want ESET, NOD, Dr Ah, Malwarebytes . . . . I would love to have a test with a comprehensive set of security products. Actually, I would love to have the budget to do that test.

    The fact is that this type of testing is expensive and time consuming. We had to make some hard choices. Our team in Japan lobbied for Sourcenext - who dominates that market, our China group wanted Rising, Eastern Europe wanted ESET, Europe wanted G-Data . . . . we have something in the range of 26 competitors that could have been included. We choose the products that felt most important to us either due to installed base or because of perception of technology that we wanted to test.

    Ideally an industry group or an independent outfit would do this - it is a big job. Anyone here willing to pitch in to pay for it?

    As for those that say the test is useless because Symantec paid for it . . . where did you get to be so cynical? o_O

    The results are valid and repeatable. pbust, yes the cloud scanning technologies worked just fine in this setup - both ours and those of Panda, McAfee . . . . To quote the report, "An HTTP replay system ensured that all target systems received the same malware as each other. It was configured
    to allow access to the internet so that products could download updates and communicate with any available ‘in the cloud’ servers."

    I'm not claiming that this is the last word in testing - but we need to get past the idea that putting a bunch of malware in a directory and scanning it gives meaningful results.
     
  16. andyman35

    andyman35 Registered Member

    Joined:
    Nov 2, 2007
    Posts:
    2,336
    Leaving aside the remarkably low score for Avira that's completely out of sync with just about every other comparitive test performed in the last 2 or 3 years,can somebody please explain the reasoning behind comparing full suites like NIS against standalone AVs such as Avira and Avast.o_O
     
  17. pbust

    pbust AV Expert

    Joined:
    Apr 29, 2009
    Posts:
    1,176
    Location:
    Spain
    Those technologies vary a lot. McAfee's for example is based on DNS queries, while ours is based on HTTP. Obviously a caching HTTP proxy in offline mode can have an effect on HTTP cloud-scanning while at the same time have no effect on DNS cloud-scanning. Can you tell me what caching proxy and config you used so we can replicate if it affects *our* cloud scanning?

    I couldn't agree more with you, really. But can you please comment on posts #14 and #15 and clarify how come a "miss" on NIS counts as "defended" while a "detection" on other engines counts as "compromise"? What am I missing here?
     
  18. ronjor

    ronjor Global Moderator

    Joined:
    Jul 21, 2003
    Posts:
    163,072
    Location:
    Texas
    I was under the impression Symantec was a member of AMTSO and as such was working hand in hand with other vendors to to improve testing procedures.

    Is your pdf file a result of of such an alliance?
     
  19. Page42

    Page42 Registered Member

    Joined:
    Jun 18, 2007
    Posts:
    6,941
    Location:
    USA
    You answered your own question. We learned our cynicism from observing companies like Symantec.
     
  20. Zimzi

    Zimzi Registered Member

    Joined:
    Jul 10, 2005
    Posts:
    289
    In the three best participants were the maximum protection settings while in others real time protection was off? :D
     
  21. the Tester

    the Tester Registered Member

    Joined:
    Jul 28, 2002
    Posts:
    2,854
    Location:
    The Gateway to the Blue Hills,WI.
    AV vendor "engaging" labs to do testing= an advertisement.
    This type of "testing" can't be taken seriously imo.
     
  22. Fajo

    Fajo Registered Member

    Joined:
    Jun 13, 2008
    Posts:
    1,814

    For once me and Page42....... Agreed! :blink:
     
  23. subset

    subset Registered Member

    Joined:
    Nov 17, 2007
    Posts:
    825
    Location:
    Austria
    Who or what is Dennis Labs o_O

    Simon Edwards writes like he is involved in Dennis Virus Labs.
    http://simonedwards.blogspot.com/2009/10/inside-dennis-virus-lab.html

    He performs in a promotional film for Symantec and performs weird promotional "tests" for Symantec.
    Oh well, he is also a member of AMTSO.

    Brave new malware testing world.
    Every vendor pays his own no name testing lab to make his product shine.
    Well, that's okay, maybe a smug self-satisfaction.
    But not of further interest for the public.

    Cheers
     
  24. Fajo

    Fajo Registered Member

    Joined:
    Jun 13, 2008
    Posts:
    1,814
    Its of interest to them simply because now they can slap another sticker on a box and people that don't know any better will go "That one has more awards lets buy it."

    GG Symantec Advertising but don't bring it here.
     
  25. subset

    subset Registered Member

    Joined:
    Nov 17, 2007
    Posts:
    825
    Location:
    Austria
    But who buys a collection of cups and trophies for his showcase just to tell everyone - Look, I'm simply fantastic.

    If Symantec were a person, I would fetch a doctor. :cautious:

    Cheers
     
Thread Status:
Not open for further replies.
  1. This site uses cookies to help personalise content, tailor your experience and to keep you logged in if you register.
    By continuing to use this site, you are consenting to our use of cookies.