This is cool and all but I didn't understand any of the results, why can't they just keep it simple? Can anyone perhaps explain which of the tools performed best? https://attackevals.mitre.org/evaluations.html
MITRE specifically stated that their objective was not to perform comparative rankings or the like. They simply test the vulnerabilities listed to determine if the security solution can first detect the activity and then mitigate it. Unlike AV labs that test against known malware attacks via malware samples, MITRE is testing using techniques deployed or could be deployed by APTs. Also, many of these attacks can be also be mitigated by OS or app patches plus manual system changes.
Well, I thought it was pretty unclear if security solutions detected the malware techniques or not. Like I said, just keep it simple, with that I mean, present the info in a clear way.
Well according to Cloudstrike, they were the most effective product: https://www.crowdstrike.com/blog/mi...owdstrike-as-the-most-effective-edr-solution/ . Of note is the best proactive detection of all products tested was only 50%. Of interest to me was Windows Defender performed much better than I expected. Unclear is this was plain WD or WD ATP. The test report would lead on to believe it was just OS based Windows Defender.
Well, that's more like it. This clearly explains how these tools performed. And BTW, it obviously was Win Def ATP that was tested. On the other hand, it still makes more sense to test real life malware against these tools. But this was more about how many malware techniques these tools are able to spot.
Per the original test report: https://attackevals.mitre.org/evaluations.html , MITRE tested both WD and WD ATP. The CloudStrike article only mentions Microsoft. Would not be surprising that CloudStrike "cherry picked" the WD test results versus the WD ATP one. Also by comparing the results for both WD and WD ATP, you have the incremental protection factor ATP provides.
I really wish there was a consumer anti-malware system that generated these kind of telemetry reports: Telemetry showing execution of Resume Viewer.exe from explorer.exe and dropping pdfhelper.cmd and autoupdate.bat • Telemetry showing write of pdfhelper.cmd • Telemetry showing write of autoupdate.bat • Telemetry showing execution of pdfhelper.cmd and update.dat • Telemetry showing execution of decoy PDF by MicrosoftPdfReader.exe • Telemetry showing Resume Viewer.exe binary and process metadata • Telemetry showing Resume Viewer.exe binary reputation • Exploit Guard audit of Resume Viewer.exe
Yes, an EDR combined with HIPS would be cool. No, they tested only enterprise security tools, this wasn't about regular AV's.
@ronjor -- Thanks for this post. It is evident that you do a lot of research in order to provide us with this sort of valuable info.
Can software vendors block a notorious criminal group’s attacks? MITRE wants to find out February 20, 2020 https://www.cyberscoop.com/fin7-mitre-fireeye/ MITRE Engenuity to Evaluate Cybersecurity Products Based on Carbanak and FIN7 Groups
MITRE Releases Results of Evaluations of 21 Cybersecurity Products April 21, 2020 https://www.mitre.org/news/press-re...s-of-evaluations-of-21-cybersecurity-products
LOTS of charts & lots of info but figuring out relative effectiveness of the various AVs/AMs is over my head.
Yes, it's completely useless, too complex to decipher. They should try to present info in a clear way, like I said before.
Using Kaspersky's evaluation for example: https://attackevals.mitre.org/APT29/results/kaspersky/ , the category to note is "None"; i.e. shown in dark blue on the chart. This amounts to a miss for the malware tactic being employed. Again, MITRE's purpose is for AV's to evaluate their protection mechanisms against known malware attack methods. It does not perform either individual or comparative rankings of test participant products. -EDIT- I will also add the MITRE Attack Threat Matrix applies to attacks against enterprise environments. That is those for the done by advanced threat actors. The chance an end user would see these would be a rare occurrence. That is not to say an end user might be hit with fileless, "living of the land," and like attacks. The difference would be how these would deployed against enterprises versus end-users. The lastest MITRE test evaluation is for tactics employed by APT29: https://attackevals.mitre.org/APT29/detection-categories.html . Malware techniques employed by the APT29 group are listed here: https://attack.mitre.org/groups/G0016/ . The prior and initial MITRE evaluation was for tactics employed by the APT3 group.
