MRG Effitas 360 Assessment & Certification Q3 2017

Discussion in 'other anti-virus software' started by anon, Dec 22, 2017.

  1. anon

    anon Registered Member

    Joined:
    Dec 27, 2012
    Posts:
    6,489
    MRG Effitas 360 Assessment & Certification Q3 2017
    2017 / 12 /12
    https://www.mrg-effitas.com/recent-projects/our-projects/
    https://www.mrg-effitas.com/wp-content/uploads/2017/12/MRG_Effitas_360_Assessment_2017_Q3-1.pdf
     
  2. plat1098

    plat1098 Guest

    Malwarebytes? Hello?
     
  3. anon

    anon Registered Member

    Joined:
    Dec 27, 2012
    Posts:
    6,489
    "Failed = Security product failed to detect all infections and remediate the system during the test procedure:
    Malwarebytes Anti-Malware
    Watchdog Anti-Malware
    Zemana Anti-Malware"
     
  4. itman

    itman Registered Member

    Joined:
    Jun 22, 2010
    Posts:
    8,122
    Location:
    U.S.A.
    Of note is Microsoft no longer participates in this comparative. Guess they didn't care for the Q2, 2017 360 comparative that showed WD's effectiveness overall and w/SmartScreen's scores enabled and disabled.
     
  5. Bill_Bright

    Bill_Bright Registered Member

    Joined:
    Jun 29, 2007
    Posts:
    3,732
    Location:
    Nebraska, USA
    Is it of note or just another biased, opportunistic bash at Microsoft? :(

    Of note is the lack of criticism for Comodo for not participating. What about BullGuard, F-Secure, K7, Sophos, G Data, ZomeAlarm, Emsisoft and others? That is not of note?

    What about Malwarebytes, Watchdog and Zemana failing? That is not of note?

    All those companies are after our money for their security products and they either failed to participate or worse, failed to pass the tests? Yet that is not of note?? :rolleyes:

    For the record, Microsoft doesn't participate because they don't program WD to score well on synthetic lab tests. Also, Microsoft is the ONLY security solution provider that does not need high synthetic test scores for marketing fodder. All the others use those scores in their marketing - evidenced by Lab logos plastered on their product boxes and websites.

    Microsoft is not in the business of selling normal consumers security suites so they don't need the marketing fodder.

    For a list of other products that lab may or may not have tested, see this Microsoft provided list of 3rd party solutions.
     
  6. daman1

    daman1 Registered Member

    Joined:
    Mar 27, 2009
    Posts:
    1,288
    Location:
    USA, MICHIGAN
    Nice :(
     
  7. itman

    itman Registered Member

    Joined:
    Jun 22, 2010
    Posts:
    8,122
    Location:
    U.S.A.
    Nice try. MRG samples on their 360 tests are all extracted from the AMTSO malware database; the same database AV-C and a number of other AV Labs use.

    If you read the Executive Summary of the report, the only thing they tried to simulate was user behavior when arriving at a web site hosting a malicious URL.
     
  8. OverDivine

    OverDivine Registered Member

    Joined:
    Jan 16, 2009
    Posts:
    24
    if uac and smartscreen were off the test is just another joke
     
  9. Bill_Bright

    Bill_Bright Registered Member

    Joined:
    Jun 29, 2007
    Posts:
    3,732
    Location:
    Nebraska, USA
    Yeah. That's pretty scary - if the tests truly represent real world scenarios their users are likely to encounter. For me, I find it "disappointing" (on an emotional level) to see what was once such a highly regarded product (Malwarebytes) go down hill. I was such a fan of Malwarebytes Antimalware and has been sad to see all its problems since V3.x came out a year ago. It's almost like watching your childhood hero fall from grace.

    Then Zemana, often touted as the go alternative to Malwarebytes, failing in the same way is disheartening too.

    Interestingly, when I try to visit the Watchdog website, Malwarebytes blocks it for using outdated or unsafe TLS security.
    Did you read it? I did. It sure does not say what you just claimed it did. :( So why did you say it did? I mean, really? :rolleyes:

    In fact, the Introduction makes it clear this test was NOT about "Real World" detection - it then gave an example of that test from way back in 2013!

    The Executive Summary says,
    Is there a better way to test these products? Probably not. But not really the point here. The point is about fairness. You are trying to obfuscate the issue, which was about biased, opportunistic Microsoft and Windows Defender bashing.

    I agree but I don't see in the report where that was the case.
     
  10. IBK

    IBK AV Expert

    Joined:
    Dec 22, 2003
    Posts:
    1,865
    Location:
    Innsbruck (Austria)
    that small database is currently of very limited use and i doubt that anyone was using it in the last months (maybe again in future if its quality and relevance improves).
     
  11. anon

    anon Registered Member

    Joined:
    Dec 27, 2012
    Posts:
    6,489
  12. Bill_Bright

    Bill_Bright Registered Member

    Joined:
    Jun 29, 2007
    Posts:
    3,732
    Location:
    Nebraska, USA
    Okay. But while Watchdog and Zemana may have family ties, Watchdog and Malwarebytes don't.

    Yeah, very small which suggests (though perhaps incorrectly) the entries were hand-picked supporting again the synthetic nature of the test.
     
  13. IBK

    IBK AV Expert

    Joined:
    Dec 22, 2003
    Posts:
    1,865
    Location:
    Innsbruck (Austria)
    I do not see where MRG states something which would suggest that "MRG samples on their 360 tests are all extracted from the AMTSO malware database" (?).
     
  14. plat1098

    plat1098 Guest

    My interest was/is chiefly regarding the vendors' putting the cash where the mouths are. I look at Malwarebytes' advertising page and it is so sad in this context. This was formerly the benchmark of antimalware applications.

    On the way other hand, Microsoft can advertise its stuff all day long as I'm concerned. If only some of the higher settings weren't so dern arcane, know what I'm saying? No matter what any comparative says, I know the fortified Home version is going to trump that in all its tweaked "glow-ry." Perhaps MRG Effitas can explain Microsoft's absence from the test situation this quarter? Speculations may not capture enough of what's trickled down to us.
     
  15. Bill_Bright

    Bill_Bright Registered Member

    Joined:
    Jun 29, 2007
    Posts:
    3,732
    Location:
    Nebraska, USA
    I suppose they have to be vague to prevent cheating. But it seems saying more than sample types used were "Trojan, Financial malware, Ransomware, Backdoor and Other" would be more helpful.
    Not sure what you mean. I look at any product's ad page and it is all marketing hype to me.
    Or just explain how those tested were selected. Otherwise, I would be curious why Emsisoft, Comodo and the others were not tested either.
    Excellent point.
     
  16. plat1098

    plat1098 Guest

    Just for Emsisoft: the relationship was apparently terminated three years ago. Insofar as the marketing hype, yes, advanced and uber-experienced will see right through it. But maybe not so many others where the mass marketing is focused.

    https://support.emsisoft.com/topic/15920-mrg-effitas-q22014-test/#entry123491

    Edit: made a correction about the amount of years elapsed. :)
     
    Last edited by a moderator: Dec 22, 2017
  17. itman

    itman Registered Member

    Joined:
    Jun 22, 2010
    Posts:
    8,122
    Location:
    U.S.A.
    I stand corrected. They have their own malware database consisting of over 300,000 malicious binaries, URLs, etc.: https://www.mrg-effitas.com/services/malware-feed/

    As far as the Q3, 2017 360 test, MRG states on page 6 of the report; 50% of the samples were legit websites hosting malicious URLs, 40% were malware samples collected from their own honeypot servers, and 10% were from fake porn sites serving up malware. 10% of the total of 351 samples used were introduced via USB from previously downloaded archives from live URLs. Of the 351 total samples used; 189 were Trojans, 30 backdoors, 80 financial malware, 50 ransomware, and 2 undisclosed type samples.
     
    Last edited: Dec 22, 2017
  18. bigwrench9

    bigwrench9 Registered Member

    Joined:
    Oct 28, 2009
    Posts:
    130
    Sheesh....Glad I stuck a fork in Malwarebytes after v2.0 :thumbd:
    So glad it's not taking up space on any of my pc's anymore.:argh:
    Just have not felt the need to run an on demand scanner alongside Kaspersky, Panda, or ESET. (no bitdefender engines allowed here) ;)
    Trialing Trend Micro on my office desktop, actually glad I grabbed a free key. Working nicely!:thumb:
     
  19. Bill_Bright

    Bill_Bright Registered Member

    Joined:
    Jun 29, 2007
    Posts:
    3,732
    Location:
    Nebraska, USA
    Well, of course that's how it is for any product. If not, advertising and marketing would not be a $200 Billion per year (in the US alone!) industry.

    Wow! That thread in the Emsisoft forum sure got out a hand.
     
  20. james246

    james246 Registered Member

    Joined:
    Nov 5, 2005
    Posts:
    138
    I actually wonder if something as simple as the free Kaspersky Anti-Ransom Tool with its blocking capabilities would embarrass many of the security products in this test.
    KAR doesn't scan or clean, it essentially just blocks bad stuff not just Ransomware, also it has an implementation of System Watcher

     
  21. Rasheed187

    Rasheed187 Registered Member

    Joined:
    Jul 10, 2004
    Posts:
    13,435
    Location:
    The Netherlands
    Malwarebytes always scores badly in MRG tests, don't know why the call themselves an "AV replacement." I really don't care if it's only a limited number of malware samples that were being used. As an AV you only have one job, that's to block malware with signatures, behavior blocking, heuristics, ML, you name it. So there is no excuse for it, I think this test is very credible, there is nothing synthetic about it. Zemana and Malwarebytes need to go back to the drawing board.

    I also think it's a bit weird that they dropped out. Perhaps because they performed horribly the last time, even with SmartScreen enabled? Actually, I don't understand how WD failed to block 19 samples, shouldn't SmartScreen block everything that's not on their white-list? Or perhaps MRG considered it a fail when SmartScreen didn't clearly label samples as malware, can't remember.
     
  22. itman

    itman Registered Member

    Joined:
    Jun 22, 2010
    Posts:
    8,122
    Location:
    U.S.A.
    In regards to the Q2 - 2017 360 test, WD + SS did poorly in the ransomware test scoring 89.8% with 5 samples missed. No surprise there since in every independent ransomware test I have viewed to date, it has likewise scored poorly against ransomware. Also in that test, WD + SS only scored 60% with 8 samples missed in PUA detection. Most likely because I believe PUA detection is not enabled by default.

    Also notable in this test was 40 samples, or 11.2%, were not blocked upon initial execution but within the 24 hr. acceptance window. This fact does draw into question SmartScreen's effectiveness as you noted. Also SmartScreen only checks executables. So any script based malware and the like would not be examined.
     
    Last edited: Dec 23, 2017
  23. Macstorm

    Macstorm Registered Member

    Joined:
    Mar 7, 2005
    Posts:
    2,631
    Location:
    Sneffels volcano
    I agree with you :thumb:
     
  24. Cutting_Edgetech

    Cutting_Edgetech Registered Member

    Joined:
    Mar 30, 2006
    Posts:
    5,601
    Location:
    USA
    Webroot missed an awful lot of samples. Detected within 24hrs is a fail to me. A lot can happen within 24 hours. The damage is already done. It's good to see that the cloud was able to identify most of the threats missed in a relatively short period of time, but it seems Webroot needs to add additional technologies to their code. I'm disappointed in seeing one of my old pals missed so many samples.
     
  25. Cutting_Edgetech

    Cutting_Edgetech Registered Member

    Joined:
    Mar 30, 2006
    Posts:
    5,601
    Location:
    USA
    Malwarebytes did very poorly on the on-demand test, decent on the ransomeware test, and not so good on the financial malware test. I think maybe they need additional coders working on their product. I may be wrong, but I think pbust stays busy working on the entire product package of Malwarebyets now, and has been doing this for the past year. I was hopping he could continue working on MBAE full time. Maybe he will have more time to work on the MBAE part of Malwarebytes now that it has been integrated into Malwarbytes. I think maybe a few additional coders would go a long way. It would be interesting to know how many full time coders they have on their staff.
     
  1. This site uses cookies to help personalise content, tailor your experience and to keep you logged in if you register.
    By continuing to use this site, you are consenting to our use of cookies.