Comparative Malware Protection Assessment

Discussion in 'other anti-virus software' started by itman, Apr 10, 2018.

  1. itman

    itman Registered Member

    Joined:
    Jun 22, 2010
    Posts:
    6,670
    Location:
    U.S.A.
    Of note is no endpoint vendor scored 100% in this test. Also the AI solutions were the lowest scorers. Cylance chose not to participate in the test.:rolleyes:

    https://www.mrg-effitas.com/wp-content/uploads/2018/04/MRG_Comparative_2018_February_report.pdf
     
  2. Minimalist

    Minimalist Registered Member

    Joined:
    Jan 6, 2014
    Posts:
    9,595
    Location:
    Slovenia
    Good results from Sophos :thumb:
    Microsoft with Defender ATP also didn't want to participate. Too bad.
     
  3. Rasheed187

    Rasheed187 Registered Member

    Joined:
    Jul 10, 2004
    Posts:
    10,773
    Location:
    The Netherlands
    Holy crap, SentinelOne and CrowdStrike performed very badly! I didn't expect this from "next gen" companies. And Symantec's behavior blocker seems to be pretty good. I also wonder why Cylance and M$ didn't want to participate, what are they so scared of? It would have been a nice chance to show of Win Def ATP. But very interesting test, thanks for posting itman. :thumb:
     
  4. itman

    itman Registered Member

    Joined:
    Jun 22, 2010
    Posts:
    6,670
    Location:
    U.S.A.
    A-V Comparatives informed me of a similar test they performed last fall. It was a commissioned test sponsored by Bitdefender: https://weblog.av-comparatives.org/advanced-endpoint-protection-test/ . In this test both Cylance and CloudStrike performed well. As such, one can only assume that testing methodology does play a factor in test results.
     
  5. Rasheed187

    Rasheed187 Registered Member

    Joined:
    Jul 10, 2004
    Posts:
    10,773
    Location:
    The Netherlands
    Thanks, will do a bit of reading, and I honestly don't know. I mean, MRG's way of testing seems to be pretty straight forward, I doubt that bad results are because of MRG's testing methodology.
     
  6. itman

    itman Registered Member

    Joined:
    Jun 22, 2010
    Posts:
    6,670
    Location:
    U.S.A.
    By "methodology," I also was referring to malware samples used and specific tests performed. For example, the A-VC test used PowerShell scripts to test fileless malware. Cylance aced this test. The reason why? It employs a script blocker.o_O So as far as its behavior detection in this area goes, its capability is unknown.
     
  7. Rasheed187

    Rasheed187 Registered Member

    Joined:
    Jul 10, 2004
    Posts:
    10,773
    Location:
    The Netherlands
    Yes, that's why Cylance was not included in the table. But very interesting results! SentinelOne and CrowdStrike have got some serious work to do. Same goes for Carbon Black. But at least they did have the guts to participate in the MRG test.

    I hope that Barkly, enSilo and Invincea will be included in future testing. And I still wonder why M$ didn't participate, I believe that based on what I've read, Win Def ATP is a pretty good product, but I hope it's not all talk, see link.

    https://cloudblogs.microsoft.com/mi...virus-is-the-most-deployed-in-the-enterprise/
     
  8. Firecat

    Firecat Registered Member

    Joined:
    Jan 2, 2005
    Posts:
    8,087
    Location:
    The land of no identity :D
    Meaning: "Hey, umm....we found that our URL blocker didn't work in this scenario, never mind that we do not really have an advanced scanner/behaviour analysis system. We fired most of our virus researchers years ago, now we have a good on-execution blocker only and nevermind that our product is useless if you do not give it access to our wide, wide servers - you should not be doing this, Symantec is good."

    For years, I have been stating that it's time this company bit the dust. That they have any market or mindshare after all the things they did in the past ten years is a miracle of sorts.
     
  9. Rasheed187

    Rasheed187 Registered Member

    Joined:
    Jul 10, 2004
    Posts:
    10,773
    Location:
    The Netherlands
    LOL, good point. BTW, I totally forgot to comment on Sophos, seems like they performed poorly in the AV-Comparatives "PowerShell-based" exploits test. I hope Mark and/or Erik Loman can explain this. Seems like they were able to block exploits on Firefox, but not when Meterpreter is launched via non-browser exploits.
     
  10. itman

    itman Registered Member

    Joined:
    Jun 22, 2010
    Posts:
    6,670
    Location:
    U.S.A.
    For reference to others, you are referring to the link posted in reply #4.

    The test was not exclusively for Powershell based attacks but also included WMI, PSExec, Task Scheduler, EternalBlue, script and other method ATP based attacks. Probably the most extensive test in this area I have seen to date. Perhaps NVT should submit OSArmor to see how it performs against these tests.
     
  11. Azure Phoenix

    Azure Phoenix Registered Member

    Joined:
    Nov 22, 2014
    Posts:
    737
    It's probably for the best to wait until the latest version is officially released before submitting it.
     
  12. Rasheed187

    Rasheed187 Registered Member

    Joined:
    Jul 10, 2004
    Posts:
    10,773
    Location:
    The Netherlands
    I'm guessing it will block quite a lot because it doesn't differentiate between malicious and normal behavior. And that's probably why some performed badly, because you can't block everything in a corporate environment. But still shocking to see that the next gen companies performed so poorly. I've read that Carbon Black and CrowdStrike are used by a lot of big corporations.
     
  13. Rasheed187

    Rasheed187 Registered Member

    Joined:
    Jul 10, 2004
    Posts:
    10,773
    Location:
    The Netherlands
    Seems like NSS Labs has done a new test, you can read more about it over at MalwareTips. Malwarebytes performed the worst, it could only block about 60% of all threats. They also always perform badly in tests done by MRG, so it's hard to take them serious anymore.

    Next gen companies like enSilo and Endgame performed pretty good. Same goes for Kaspersky and Bitdefender. But seems to be a very exciting industry, the only way to proof that you're any good is to participate in these kind of tests. And remember the beef between Cylance and Sophos, this was entertainment at its best LOL. The third link is another test that was aced by enSilo.

    https://www.nsslabs.com/company/new...anced-endpoint-protection-group-test-results/
    https://www.bankinfosecurity.com/blogs/av-wars-sophos-vs-cylance-p-2172
    https://www.av-test.org/en/antiviru...ws-10/february-2018/ensilo-ensilo-2.7-180603/
     
  14. Rasheed187

    Rasheed187 Registered Member

    Joined:
    Jul 10, 2004
    Posts:
    10,773
    Location:
    The Netherlands
    BTW, do you understand why NSS Labs has beef with CrowdStrike? Last year (2017) they had major beef and in the 2018 test they were once again mentioned with a "cautious" rating, see first link. I wonder why they have difficulties testing it. And also fun to see that Trend Micro is taking aim at "next generation" endpoint vendors.

    https://www.sentinelone.com/blog/sentinelone-top-performer-nss-labs-security-value-map-2018/
    https://www.zdnet.com/article/crowd...-legal-challenge-against-subversive-nss-labs/
    https://blog.trendmicro.com/endpoint-security-testing-matters-new-nss-aep-test-results/
     
  15. itman

    itman Registered Member

    Joined:
    Jun 22, 2010
    Posts:
    6,670
    Location:
    U.S.A.
    Actually, its the other way around. It's CloudStrike that "has a beef" with NSS Labs testing. And they are not the only AV vendor in this category.

    2017 was the beginning of the "new and controversial" use of simulated malware to accommodate NextGen vendors complaints that conventional AV lab tests did not fairly portray their malware protection capability. And there was a far worse "culprit" in this simulated malware charade - AV-Test who deployed 100% simulated malware in their testing of NextGen products. At least, NSS Labs "drew a line" on what type and the use extent of simulated malware in the 2017 test which resulted in Cylance "bad mouthing" their test procedures.

    Of note is the use of simulated malware is nothing new. MRG has included a few of them in its testing for some time. The major and important difference is MRG never used simulated malware results in its overall protection scoring of tested security products; only for informational purposes.

    Bottom line - simulated malware has no business being used in conventional AV security product testing. It is appropriate for penetration testing scenarios and needs to be restricted to that context.
     
  16. itman

    itman Registered Member

    Joined:
    Jun 22, 2010
    Posts:
    6,670
    Location:
    U.S.A.
  17. itman

    itman Registered Member

    Joined:
    Jun 22, 2010
    Posts:
    6,670
    Location:
    U.S.A.
    In regards to the 2018 NSS Labs Endpoint Comparative, Eset's Corporate blog has an article on it from which I am posting a few excerpts:
    https://www.eset.com/us/about/newsr...bs-advanced-endpoint-protection-test-results/

    Again, I urge folks to read the above Virus Bulletin article on just how bad things have become on the AV Lab test scene.
     
  18. Rasheed187

    Rasheed187 Registered Member

    Joined:
    Jul 10, 2004
    Posts:
    10,773
    Location:
    The Netherlands
    But the thing is, they don't have any problems testing all of the other tools, except for CrowdStrike. If they couldn't measure the effectiveness, I wonder why they even mentioned them and placed them last one the list. Also, this test wasn't sponsored, so you would think they are not biased.

    That's interesting because Eset actually performed pretty good. BTW, seems like Secdo is another interesting tool, it could spot in-memory ransomware, while others missed it. And in the second link you can download a copy of the NSS Labs test report of FortiClient, too bad there is no technical info to be found.

    https://blog.secdo.com/multiple-groups-exploiting-eternalblue-weeks-before-wannacry
    https://www.fortinet.com/products/endpoint-security/forticlient.html (scroll to Certifications)
     
  19. itman

    itman Registered Member

    Joined:
    Jun 22, 2010
    Posts:
    6,670
    Location:
    U.S.A.
    Cloudstrike declined to be tested in the 2018 comparative; reasons are fairly obvious. I believe NSS Labs mentioned them since they were included in the 2017 comparative.
     
  20. itman

    itman Registered Member

    Joined:
    Jun 22, 2010
    Posts:
    6,670
    Location:
    U.S.A.
    You have to buy the test report from NSS Labs. This is their primary revenue source BTW.
     
    Last edited: May 19, 2018
  21. Rasheed187

    Rasheed187 Registered Member

    Joined:
    Jul 10, 2004
    Posts:
    10,773
    Location:
    The Netherlands
    Actually, I was wrong. They do provide some more info about what exactly is tested in the FortiClient report and I believe this is the full report. But would be nice to know what type of malware was tested.
     
Loading...
  1. This site uses cookies to help personalise content, tailor your experience and to keep you logged in if you register.
    By continuing to use this site, you are consenting to our use of cookies.