MITRE Changes the Game in Security Product Testing

This is cool and all but I didn't understand any of the results, why can't they just keep it simple? Can anyone perhaps explain which of the tools performed best?

https://attackevals.mitre.org/evaluations.html

 

MITRE specifically stated that their objective was not to perform comparative rankings or the like. They simply test the vulnerabilities listed to determine if the security solution can first detect the activity and then mitigate it.

Unlike AV labs that test against known malware attacks via malware samples, MITRE is testing using techniques deployed or could be deployed by APTs. Also, many of these attacks can be also be mitigated by OS or app patches plus manual system changes.

 

Well, I thought it was pretty unclear if security solutions detected the malware techniques or not. Like I said, just keep it simple, with that I mean, present the info in a clear way.

 

Well according to Cloudstrike, they were the most effective product: https://www.crowdstrike.com/blog/mi...owdstrike-as-the-most-effective-edr-solution/ .

Of note is the best proactive detection of all products tested was only 50%.

Of interest to me was Windows Defender performed much better than I expected. Unclear is this was plain WD or WD ATP. The test report would lead on to believe it was just OS based Windows Defender.

 

Well, that's more like it. This clearly explains how these tools performed. And BTW, it obviously was Win Def ATP that was tested. On the other hand, it still makes more sense to test real life malware against these tools. But this was more about how many malware techniques these tools are able to spot.

 

Per the original test report: https://attackevals.mitre.org/evaluations.html , MITRE tested both WD and WD ATP. The CloudStrike article only mentions Microsoft. Would not be surprising that CloudStrike "cherry picked" the WD test results versus the WD ATP one.

Also by comparing the results for both WD and WD ATP, you have the incremental protection factor ATP provides.

 

I really wish there was a consumer anti-malware system that generated these kind of telemetry reports:

Telemetry showing execution of Resume Viewer.exe from explorer.exe and dropping pdfhelper.cmd and autoupdate.bat
• Telemetry showing write of pdfhelper.cmd
• Telemetry showing write of autoupdate.bat
• Telemetry showing execution of pdfhelper.cmd and update.dat
• Telemetry showing execution of decoy PDF by MicrosoftPdfReader.exe
• Telemetry showing Resume Viewer.exe binary and process metadata
• Telemetry showing Resume Viewer.exe binary reputation
• Exploit Guard audit of Resume Viewer.exe

 

Yes, an EDR combined with HIPS would be cool.

No, they tested only enterprise security tools, this wasn't about regular AV's.

 

@ronjor -- Thanks for this post. It is evident that you do a lot of research in order to provide us with this sort of valuable info.