MITRE Changes the Game in Security Product Testing

Discussion in 'other anti-malware software' started by ronjor, Nov 29, 2018.

  1. ronjor

    ronjor Global Moderator

    Joined:
    Jul 21, 2003
    Posts:
    68,565
    Location:
    Texas
    Kelly Jackson Higgins 11/29/2018
     
  2. Rasheed187

    Rasheed187 Registered Member

    Joined:
    Jul 10, 2004
    Posts:
    11,250
    Location:
    The Netherlands
  3. itman

    itman Registered Member

    Joined:
    Jun 22, 2010
    Posts:
    7,139
    Location:
    U.S.A.
    MITRE specifically stated that their objective was not to perform comparative rankings or the like. They simply test the vulnerabilities listed to determine if the security solution can first detect the activity and then mitigate it.

    Unlike AV labs that test against known malware attacks via malware samples, MITRE is testing using techniques deployed or could be deployed by APTs. Also, many of these attacks can be also be mitigated by OS or app patches plus manual system changes.
     
  4. Rasheed187

    Rasheed187 Registered Member

    Joined:
    Jul 10, 2004
    Posts:
    11,250
    Location:
    The Netherlands
    Well, I thought it was pretty unclear if security solutions detected the malware techniques or not. Like I said, just keep it simple, with that I mean, present the info in a clear way.
     
  5. itman

    itman Registered Member

    Joined:
    Jun 22, 2010
    Posts:
    7,139
    Location:
    U.S.A.
    Well according to Cloudstrike, they were the most effective product: https://www.crowdstrike.com/blog/mi...owdstrike-as-the-most-effective-edr-solution/ .
    Of note is the best proactive detection of all products tested was only 50%.

    Of interest to me was Windows Defender performed much better than I expected. Unclear is this was plain WD or WD ATP. The test report would lead on to believe it was just OS based Windows Defender.
     
    Last edited: Dec 10, 2018
  6. Rasheed187

    Rasheed187 Registered Member

    Joined:
    Jul 10, 2004
    Posts:
    11,250
    Location:
    The Netherlands
    Well, that's more like it. This clearly explains how these tools performed. And BTW, it obviously was Win Def ATP that was tested. On the other hand, it still makes more sense to test real life malware against these tools. But this was more about how many malware techniques these tools are able to spot.
     
  7. itman

    itman Registered Member

    Joined:
    Jun 22, 2010
    Posts:
    7,139
    Location:
    U.S.A.
    Per the original test report: https://attackevals.mitre.org/evaluations.html , MITRE tested both WD and WD ATP. The CloudStrike article only mentions Microsoft. Would not be surprising that CloudStrike "cherry picked" the WD test results versus the WD ATP one.

    Also by comparing the results for both WD and WD ATP, you have the incremental protection factor ATP provides.
     
    Last edited: Dec 15, 2018
  8. Gein

    Gein Registered Member

    Joined:
    Dec 8, 2013
    Posts:
    152
    I really wish there was a consumer anti-malware system that generated these kind of telemetry reports:

    Telemetry showing execution of Resume Viewer.exe from explorer.exe and dropping pdfhelper.cmd and autoupdate.bat
    • Telemetry showing write of pdfhelper.cmd
    • Telemetry showing write of autoupdate.bat
    • Telemetry showing execution of pdfhelper.cmd and update.dat
    • Telemetry showing execution of decoy PDF by MicrosoftPdfReader.exe
    • Telemetry showing Resume Viewer.exe binary and process metadata
    • Telemetry showing Resume Viewer.exe binary reputation
    • Exploit Guard audit of Resume Viewer.exe
     
  9. Rasheed187

    Rasheed187 Registered Member

    Joined:
    Jul 10, 2004
    Posts:
    11,250
    Location:
    The Netherlands
    Yes, an EDR combined with HIPS would be cool.

    No, they tested only enterprise security tools, this wasn't about regular AV's.
     
Loading...
  1. This site uses cookies to help personalise content, tailor your experience and to keep you logged in if you register.
    By continuing to use this site, you are consenting to our use of cookies.