Of note is no endpoint vendor scored 100% in this test. Also the AI solutions were the lowest scorers. Cylance chose not to participate in the test. https://www.mrg-effitas.com/wp-content/uploads/2018/04/MRG_Comparative_2018_February_report.pdf
Holy crap, SentinelOne and CrowdStrike performed very badly! I didn't expect this from "next gen" companies. And Symantec's behavior blocker seems to be pretty good. I also wonder why Cylance and M$ didn't want to participate, what are they so scared of? It would have been a nice chance to show of Win Def ATP. But very interesting test, thanks for posting itman.
A-V Comparatives informed me of a similar test they performed last fall. It was a commissioned test sponsored by Bitdefender: https://weblog.av-comparatives.org/advanced-endpoint-protection-test/ . In this test both Cylance and CloudStrike performed well. As such, one can only assume that testing methodology does play a factor in test results.
Thanks, will do a bit of reading, and I honestly don't know. I mean, MRG's way of testing seems to be pretty straight forward, I doubt that bad results are because of MRG's testing methodology.
By "methodology," I also was referring to malware samples used and specific tests performed. For example, the A-VC test used PowerShell scripts to test fileless malware. Cylance aced this test. The reason why? It employs a script blocker. So as far as its behavior detection in this area goes, its capability is unknown.
Yes, that's why Cylance was not included in the table. But very interesting results! SentinelOne and CrowdStrike have got some serious work to do. Same goes for Carbon Black. But at least they did have the guts to participate in the MRG test. I hope that Barkly, enSilo and Invincea will be included in future testing. And I still wonder why M$ didn't participate, I believe that based on what I've read, Win Def ATP is a pretty good product, but I hope it's not all talk, see link. https://cloudblogs.microsoft.com/mi...virus-is-the-most-deployed-in-the-enterprise/
Meaning: "Hey, umm....we found that our URL blocker didn't work in this scenario, never mind that we do not really have an advanced scanner/behaviour analysis system. We fired most of our virus researchers years ago, now we have a good on-execution blocker only and nevermind that our product is useless if you do not give it access to our wide, wide servers - you should not be doing this, Symantec is good." For years, I have been stating that it's time this company bit the dust. That they have any market or mindshare after all the things they did in the past ten years is a miracle of sorts.
LOL, good point. BTW, I totally forgot to comment on Sophos, seems like they performed poorly in the AV-Comparatives "PowerShell-based" exploits test. I hope Mark and/or Erik Loman can explain this. Seems like they were able to block exploits on Firefox, but not when Meterpreter is launched via non-browser exploits.
For reference to others, you are referring to the link posted in reply #4. The test was not exclusively for Powershell based attacks but also included WMI, PSExec, Task Scheduler, EternalBlue, script and other method ATP based attacks. Probably the most extensive test in this area I have seen to date. Perhaps NVT should submit OSArmor to see how it performs against these tests.
It's probably for the best to wait until the latest version is officially released before submitting it.
I'm guessing it will block quite a lot because it doesn't differentiate between malicious and normal behavior. And that's probably why some performed badly, because you can't block everything in a corporate environment. But still shocking to see that the next gen companies performed so poorly. I've read that Carbon Black and CrowdStrike are used by a lot of big corporations.
Seems like NSS Labs has done a new test, you can read more about it over at MalwareTips. Malwarebytes performed the worst, it could only block about 60% of all threats. They also always perform badly in tests done by MRG, so it's hard to take them serious anymore. Next gen companies like enSilo and Endgame performed pretty good. Same goes for Kaspersky and Bitdefender. But seems to be a very exciting industry, the only way to proof that you're any good is to participate in these kind of tests. And remember the beef between Cylance and Sophos, this was entertainment at its best LOL. The third link is another test that was aced by enSilo. https://www.nsslabs.com/company/new...anced-endpoint-protection-group-test-results/ https://www.bankinfosecurity.com/blogs/av-wars-sophos-vs-cylance-p-2172 https://www.av-test.org/en/antiviru...ws-10/february-2018/ensilo-ensilo-2.7-180603/
BTW, do you understand why NSS Labs has beef with CrowdStrike? Last year (2017) they had major beef and in the 2018 test they were once again mentioned with a "cautious" rating, see first link. I wonder why they have difficulties testing it. And also fun to see that Trend Micro is taking aim at "next generation" endpoint vendors. https://www.sentinelone.com/blog/sentinelone-top-performer-nss-labs-security-value-map-2018/ https://www.zdnet.com/article/crowd...-legal-challenge-against-subversive-nss-labs/ https://blog.trendmicro.com/endpoint-security-testing-matters-new-nss-aep-test-results/
Actually, its the other way around. It's CloudStrike that "has a beef" with NSS Labs testing. And they are not the only AV vendor in this category. 2017 was the beginning of the "new and controversial" use of simulated malware to accommodate NextGen vendors complaints that conventional AV lab tests did not fairly portray their malware protection capability. And there was a far worse "culprit" in this simulated malware charade - AV-Test who deployed 100% simulated malware in their testing of NextGen products. At least, NSS Labs "drew a line" on what type and the use extent of simulated malware in the 2017 test which resulted in Cylance "bad mouthing" their test procedures. Of note is the use of simulated malware is nothing new. MRG has included a few of them in its testing for some time. The major and important difference is MRG never used simulated malware results in its overall protection scoring of tested security products; only for informational purposes. Bottom line - simulated malware has no business being used in conventional AV security product testing. It is appropriate for penetration testing scenarios and needs to be restricted to that context.
In regards to the AV lab testing "debacles" that started in 2017, this Virus Bulletin publication is a must read: https://www.virusbulletin.com/virusbulletin/2017/11/vb2017-paper-testing-world-turned-upside-down/
In regards to the 2018 NSS Labs Endpoint Comparative, Eset's Corporate blog has an article on it from which I am posting a few excerpts: https://www.eset.com/us/about/newsr...bs-advanced-endpoint-protection-test-results/ Again, I urge folks to read the above Virus Bulletin article on just how bad things have become on the AV Lab test scene.
But the thing is, they don't have any problems testing all of the other tools, except for CrowdStrike. If they couldn't measure the effectiveness, I wonder why they even mentioned them and placed them last one the list. Also, this test wasn't sponsored, so you would think they are not biased. That's interesting because Eset actually performed pretty good. BTW, seems like Secdo is another interesting tool, it could spot in-memory ransomware, while others missed it. And in the second link you can download a copy of the NSS Labs test report of FortiClient, too bad there is no technical info to be found. https://blog.secdo.com/multiple-groups-exploiting-eternalblue-weeks-before-wannacry https://www.fortinet.com/products/endpoint-security/forticlient.html (scroll to Certifications)
Cloudstrike declined to be tested in the 2018 comparative; reasons are fairly obvious. I believe NSS Labs mentioned them since they were included in the 2017 comparative.
Actually, I was wrong. They do provide some more info about what exactly is tested in the FortiClient report and I believe this is the full report. But would be nice to know what type of malware was tested.
