MRG Effitas 360 Assessment & Certification Programme Q4 -2014

Discussion in 'other anti-malware software' started by malexous, Feb 3, 2015.

  1. malexous

    malexous Registered Member

    Joined:
    Jun 18, 2010
    Posts:
    828
    Location:
    Ireland
    https://www.mrg-effitas.com/mrg-effitas-360-assessment-certification-programme-q4-2014/

    Tested:
    avast! Internet Security 2015.10.0.2208.712
    Avira Internet Security 2014 14.0.7.468
    BitDefender Internet Security 2015 18.20.0.1429
    ESET Smart Security 8.0.304.0
    Kaspersky Internet Security 2015 15.0.1.415.0.598
    Malwarebytes Anti-Malware (Premium)2.0.4.1028
    McAfee Internet Security 2015
    Microsoft Security Essentials 4.7.0202
    Norton Security 2015 22.1.0.9
    Panda Internet Security 2015 15.0.4
    SurfRight HitmanPro 3.7.9 - Build 232*
    Trend Micro Internet Security 2015 8.0.1133
    VIPRE Internet Security 2015 8.0.5.3
    Webroot SecureAnywhere Internet Security 8.0.6.28
     
  2. Minimalist

    Minimalist Registered Member

    Joined:
    Jan 6, 2014
    Posts:
    5,043
    Thank you for sharing. Interesting combination of AVs got certified :)
     
  3. Rompin Raider

    Rompin Raider Registered Member

    Joined:
    May 6, 2010
    Posts:
    1,228
    Location:
    North Texas
    Thanks malexous...pretty consistent with previous tests.
     
  4. Mayahana

    Mayahana Banned

    Joined:
    Sep 13, 2014
    Posts:
    2,220
    But not consistent with tests from anyone else.. I'm pretty suspicious about these results, how did they manage to break products with DNA and/or reputation engines that every other firm has pulling 100% detection?
     
  5. Thankful

    Thankful Savings Monitor

    Joined:
    Feb 28, 2005
    Posts:
    3,731
    Location:
    New York City
    Good job Eset, Webroot !
     
  6. DX2

    DX2 Guest

    Surprise Surprise..
     
  7. Charyb

    Charyb Registered Member

    Joined:
    Jan 16, 2013
    Posts:
    552
    Great job Webroot. I'll drink to that.
     
  8. SweX

    SweX Registered Member

    Joined:
    Apr 21, 2007
    Posts:
    6,429
    :thumb:
     
  9. Frank the Perv

    Frank the Perv Banned

    Joined:
    Dec 16, 2005
    Posts:
    882
    Location:
    Virginia, USA
    That's funny. That's really funny. There is a Kool-Aid reference here for those who missed it.


    I don't bash tests of the major and known testing organizations. They do a good job overall. Yes, there are always testing limitations.

    Kaspersky (1st Place) is great as usual.

    ESET (2nd Place) did surprisingly well. Bravo ESET. I hope they continue to do well.

    Bitdefender (3rd Place) -- is a perpetual top finisher.

    Webroot (4th) did very surprisingly well -- as indicated by the reactions in this thread.

    And Symantec, Trend, and Avira followed closely behind.

    As always... is this a trend, or just a random data point? Time will tell.

    I do like it that we are getting a diversity of top finishers in different tests. Competition is a good thing.

    And... not to overlook Hitman Pro -- the Hitman dudes are doing a great job. If I understand it correctly (and I have not read the fine print), Hitman Pro beat every AV except Kaspersky. That's pretty darn good.

    So, it you had MSE (last place finisher) + Hitman Pro*, you would have beat everybody except Kaspersky. And that is assuming redundant detection for those two products. It could be better than Kaspersky. Something to think about.
    (Again, I have not read the fine print, it this is not correct, please correct me).


    *HitmanPro does not have real-time protection



    -Frank
     
    Last edited: Feb 4, 2015
  10. Mayahana

    Mayahana Banned

    Joined:
    Sep 13, 2014
    Posts:
    2,220
    I just wonder how they slipped so many samples by Trend's DNA+Reputation, and Norton's Reputation. These are anomalies I am perplexed about. It's pretty cut and dry - either Trend has the DNA in their 700,000,000 DNA database or they don't. Either the file has a good reputation, or it doesn't. This is the first test of Trend 2015 I am aware of where it didn't score the best, or very close to the best. So color me perplexed!

    But let's not forget this finer detail... The differences in score may amount to a SINGLE DETECTION. Their chart is a bit misleading, and accentuates differences that are miniscule. Why? For example Symantec and Trend both scored 97.9%. Webroot at 98.1%, a difference of a mere 0.2%.. Without more details, what's that amount to? A single file difference? Panda looks bad, but it's actually at 97%, or a mere .3-.5% behind everyone else.

    But I believe this is the most important take away;

    Also remember they tossed in a couple dozen PUP's. Were optional PUP detections enabled? I doubt it! If you tossed Trend into Hypersensitive, with PUP checkbox on, that MAY account for the variance. Norton has never been good with PUP's unless you set it to aggressive for Low-Risk, does that also account for the variances? My math tells me Webroot missed 10 samples, Trend Missed 11. The list doesn't look quite the same now, does it? Especially considering all of the detection differences may simply be the result of unchecked PUP settings with products that are not known for aggressive PUP detection without the checkbox! (Avast, Norton, Trend, etc) While products with strong PUP detection (ESET) scored at the top.
     
    Last edited: Feb 4, 2015
  11. Frank the Perv

    Frank the Perv Banned

    Joined:
    Dec 16, 2005
    Posts:
    882
    Location:
    Virginia, USA

    Agree. I'm gonna guess this is an outlier data point. Trend has been on a steep incline in performance and detection ever since the 2015 product was released.

    I'm gonna try Trend again one of these days soon..
     
  12. Mayahana

    Mayahana Banned

    Joined:
    Sep 13, 2014
    Posts:
    2,220
    I think the variance may be entirely to blame on PUP's without pup options enabled on products that default on anemic pup detection without those optional settings. Tests with actual malware only, Trend has been pulling incredible numbers. But if you don't toss Trend into Hypersenstive, with the PUP checkbox, it's going to let them through. I find similar things with Norton (Low Risk Dialed up fixes it), and Avast (Pup checkbox enabled, high sensitivity, reputation checkbox)
     
  13. Thankful

    Thankful Savings Monitor

    Joined:
    Feb 28, 2005
    Posts:
    3,731
    Location:
    New York City
    The PUA option must be available during the installation process for it to be selected.
    From the report:
    "During the installation of the security application, if an option to detect PUAs is given, it is selected."
     
    Last edited: Feb 4, 2015
  14. Mayahana

    Mayahana Banned

    Joined:
    Sep 13, 2014
    Posts:
    2,220
    Sadly, that option is not on Avast, Trend, or Symantec. But it IS on some of the other ones that scored well.
     
  15. Charyb

    Charyb Registered Member

    Joined:
    Jan 16, 2013
    Posts:
    552
    I don't think the option is given at installation.

    As for setting Trend to detect PUPs, it's not clear within the GUI of what to enable for PUP detection.
     
  16. Mayahana

    Mayahana Banned

    Joined:
    Sep 13, 2014
    Posts:
    2,220
    I think this new hosted business Avast needs to deploy with PUP enabled. For Norton 2015, you need to put it on Aggressive, and Low Risk to "Always". In my experience if you do not it will let some pups through. It's not well documented that Trend really should be placed into Hypersensitive, I wish it was a default setting, and I wish they would default the detection slider for DNA/Fingerprinting to maximum, right now it defaults to 'medium'. I bet Emsisoft would have scored extremely well on this test do to their default aggressive anti-pup stance.
     
  17. Charyb

    Charyb Registered Member

    Joined:
    Jan 16, 2013
    Posts:
    552
    Did they leave one out? Misprint?
     

    Attached Files:

  18. Krusty

    Krusty Registered Member

    Joined:
    Feb 3, 2012
    Posts:
    2,865
    Location:
    Australia
    Hi Mayahana,

    Where in Norton is the Low Risk > Always setting?

    Thanks,
    Krusty

    Edit: In fact the 2015 vesrion of NS / NSwB does not have a setting for PUA detection like previous versions, at least that I can find.
     
    Last edited: Feb 4, 2015
  19. Mayahana

    Mayahana Banned

    Joined:
    Sep 13, 2014
    Posts:
    2,220
    Good catch, and sloppy. It makes me wonder even more.
     
  20. Mayahana

    Mayahana Banned

    Joined:
    Sep 13, 2014
    Posts:
    2,220
    I'd have to look. Some of the settings for aggressive impact PUA, and there was a Low-Risk setting somewhere. I just don't have it loaded right now to check.
     
  21. Krusty

    Krusty Registered Member

    Joined:
    Feb 3, 2012
    Posts:
    2,865
    Location:
    Australia
    Sorry to everyone for going off topic.

    The settings for low risks are Ask Me (Default), Remove and Ignore.

    The three settings you can change to Aggressive are Boot Time Protection, SONAR and Heuristics. When you change Heuristics to Aggressive you get a warning, but I haven't had any false positives.

    Cheers.
     
  22. Frank the Perv

    Frank the Perv Banned

    Joined:
    Dec 16, 2005
    Posts:
    882
    Location:
    Virginia, USA

    Ah hah. A lack of attention to detail.

    Additionally, they state that 15 products are tested. Then in the list of "Security Applications Tested" on page 6, they only list 14 -- leaving out ThreatTrack (whatever that is).

    And.. to pile on MRG a little bit more, the CEO seems a little flaky. I was not going to bring this up... But what the heck. MRG Chris (CEO) was banned from Wilders... for good reason. MRG Chris seems like a buffoon.

    See Wilders Thread:
    https://www.wilderssecurity.com/threads/mrg-effitas-360-assessment-q3-2014.371574/#post-2441914

    ..and look at the links that FleischmannTV posted.

    Is the Charyb in this thread the same as the one in the Emsisoft forums? If so -- great job standing up to the bully. And SweX in this thread appears in the referenced thread also.


    All that stated, I do not think the tests were scammed. Although, when one reads MRG Chris's statement -- "We wish Emsisoft well for the future and hope they can find a lab which consistently places their product at or near the top of their tests - as they did with us."

    Hmmmmm.

    Poor sentence structure and word choice aside -- when statements like this are made, I can definitely see where some people could form conclusions that things at MRG are not necessarily consistent with pure testing integrity.


    All that stated, I do think the MRG tests are legit... for a number of reasons.
     
    Last edited: Feb 4, 2015
  23. Thankful

    Thankful Savings Monitor

    Joined:
    Feb 28, 2005
    Posts:
    3,731
    Location:
    New York City
    MRG runs their tests from the point of view of an ordinary, unsophisticated user, not a professional tester.
    Programs that require tweaking to catch certain malware will not score as well as those where default
    settings are more likely to provide additional protection.
     
  24. fax

    fax Registered Member

    Joined:
    May 30, 2005
    Posts:
    3,723
    Location:
    localhost
    If this is the case, it would be enough for MRG to report the type of infection for the failed cases. This is rather important to be able to judge how far a security tool can defend a system from serious infections like ramsoware or financial malware as compared to just annoying PUPs.
     
  25. itman

    itman Registered Member

    Joined:
    Jun 22, 2010
    Posts:
    2,969
    Location:
    U.S.A.
    MRG results don't sync with results with the AV labs; VB100, AV- Comparatives, and AV-test to name a few. I would stick with the AV lab tests.