MRG Effitas – Real World Exploit Prevention – March 2015 (sponsored by Surfright)

Discussion in 'other anti-malware software' started by FleischmannTV, Apr 7, 2015.

  1. itman

    itman Registered Member

    Joined:
    Jun 22, 2010
    Posts:
    8,111
    Location:
    U.S.A.
    According to A-V Comparatives, their Real World Comparative does include exploit testing: http://www.av-comparatives.org/wp-content/uploads/2014/10/avc_factsheet2014_09.pdf .

    So this might be the closest to an objective test that can be currently had. Also their results are significantly different than that given for the recent MRG report.

    The thing I find very suspicious in the MRG report is Eset's Smart Security results; they just don't correspond with any other testing of the product. Also it could be very well that the archaic system configuration used in many of the MRG tests are just not supported and correspondingly protected by many of the main stream vendors anymore.

    Finally in the MRG "artificial" exploit test which one has to strongly suspect was custom engineered for HMPA, only EMET and MBAE additionally blocked anything. Plus the test was done using FireFox exploiting a plug-in. I would love to see how EMET would perform against a similar exploit using IE.
     
    Last edited: Apr 9, 2015
  2. Gapliin

    Gapliin Registered Member

    Joined:
    Feb 12, 2012
    Posts:
    81
    While we're at it: @Zoltan_MRG, there's a typo in the version number of Kaspersky Internet Security. You obviously mean 15.0.1.415 and not 5.0.1.415. (Pages 30, 33 and 36)

    And btw, since the text says all the test were carried out between 9th March and 27th March, 2015: version 15.0.2.361 has already been released since 3rd February, 2015. Not sure why you were using an older version.
     
  3. Rasheed187

    Rasheed187 Registered Member

    Joined:
    Jul 10, 2004
    Posts:
    13,412
    Location:
    The Netherlands
    I don't understand all the hype about this test, I don't see any great surprises. I also think it's fair of SurfRight to show that an older version of HMPA was not perfect. MBAE also gets a high score, and according to pbust some of the misses were fixed.

    EMET probably performs less good, perhaps it lacks a couple of blocking techniques. And I have not read the whole report yet, but I wonder how the AV's blocked the exploits, was it with HIPS or signature? I only care about pro active detection (behavior blocking).
     
  4. Frank the Perv

    Frank the Perv Banned

    Joined:
    Dec 16, 2005
    Posts:
    882
    Location:
    Virginia, USA
    Then you probably don't understand the nature of testing.

    Testing is supposed to be impartial, objective, and fair to all those tested.


    Thank you.

    -Frank
     
  5. Rasheed187

    Rasheed187 Registered Member

    Joined:
    Jul 10, 2004
    Posts:
    13,412
    Location:
    The Netherlands
    Yes, I think we all agree with that, but how many of these "sponsored" tests are perfect and completely fair? Like I said, I didn't see anything shocking, it's what I expected. Y'all acting like some products where seriously disadvantaged or favored.

    And perhaps you guys are right about some stuff, but I like to focus on the big picture. Both HMPA and MBAE are good choices when you want to run a dedicated exploit blocker, that's the end conclusion.
     
  6. Gapliin

    Gapliin Registered Member

    Joined:
    Feb 12, 2012
    Posts:
    81
    The AV's didn't block them with HIPS or behaviour blocking. MRG only monitored if a new process has been started or loaded and according to the rules they've chosen it's game-over after this. So every warning/alert which happend after malware execution was ignored. So the test only was checking for how good the products blocked the exploits. But...
    Some products were included which never state they have any exploit-blocking features included. If the whole point is to test their exploit-blocking capability this simply makes no sense. Why compare them to software like MBAE, HMPA or EMET then?
    • Either you let those antimalware-suites use all their protection modules (sandboxing, HIPS, behaviour blocking, etc. - everything that comes in after malware execution) and compare them of their value in protecting the user from the ultimate evil, the payload.
    • Or you only include products which are able to detect exploits and block/prevent their execution.
    Everything else is just comparing apples and oranges.
     
    Last edited: Apr 9, 2015
  7. fax

    fax Registered Member

    Joined:
    May 30, 2005
    Posts:
    3,899
    Location:
    localhost
    Lesson learned from this test? That next time Surfright should subcontract another laboratory with a more critical review on the testing methodology and better reputation. More fair, robust and transparent results and a typical win-win situation (product is good and testing organisation gets visibility). Instead now this look like a lose-lose situation with a product that is artificially pushed up and a testing organisation that managed the test superficially… Not bad, they miss two birds with a stone :D
     
  8. 142395

    142395 Guest

    MBAE definitely adds meaningful protection. Norton's exploit protection heavily rely on IPS and browser protection (Generic Exploit Blocking technology). Tho browser protection may be able to block some unknown 0day, basically they're only effective against exploit against known vuln. Norton rely on UxP and application behavior lockdown against unknown 0day exploit, but I don't expect too much for them. To be honest, I don't know how effective they are as I can't test them―Norton always block exploits before they chime in by IPS or browser protection, so I need to generate real 0day to test which is beyond my expertise. Anyway, browser protection and lockdown only works against popular application (former for IE, Fx, Chrome. latter is problem, we don't know what apps are protected by lockdown) so if you use unpopular application, add MBAE protection for them.

    Also note, Norton is not much effective about document scan. When document includes exploit, Norton is not much effective and this is why I use OfficeMalScanner (I bit care about its no update since end of 2013 tho). Not only based on my hand on testing, but I found some academic research paper about document exploit detection, and it confirmed my finding of Norton is not capable about detecting document exploit. To be fair, in my testing Norton properly blocked subsequent drive-by by either URL blocking or download insight when link is alive.
    Sorry for nitpicking but that's not exact. Correct sequence is:
    URL filtration>IPS>signature and heuristic scan>download insight>SONAR
    In this context reputation is equal to download insight. I don't know what DNA means but if it means cloud classification by machine learning, it should be included in heuristic scan (Suspicious.cloud) and DL insight.
     
    Last edited by a moderator: Apr 10, 2015
  9. 142395

    142395 Guest

    I know, and while they may all stops 9mm bullet (well, it's simplified example, I know too much about those military things...) and broken by .50 magnum or 5.56mm NATO rifle, still if one of them can block .44 magnum or even .357 magnum (9mm) still it have value. I want to know which is the hardest to bypass.
    Did you correct Office 20010 bug in past report which I posted in a past MRG test thread?
    AVC test surely includes exploit, but it also includes many non-exploit executalbes so can't be viewed as exploit test. Also do not speak about correspondence, when methodology differ, it's natural that they don't correspond and most often such difference can be reasonably explained if you know technical deteals about the products and carefully read methodology (tho I haven't seen really detailed methodology in any tests).
    Only EMET have EAF+, tho HMPA have IAF, dynamic heap spray, and HW-assisted CGI and both of MBAE and HMPA adds StackExec and behavior protection. As HMPA & MBAE can add behavior protection when new bypass or new type of exploit (e.g. logic flaw) come, EMET also can add EAF+ module specification and ASR which I find useful. Like Kees, I also added some custom modules. Note a researcher who recently bypassed EMET and MBAE said EMET is much tougher than MBAE.

    Both, some AV have NIPS (sig) and behavior-based anti-exploit (e.g. Norton UxP, Kaspersky AEP, F-Secure Deepguard 6.0, ESET exploit blocker, G-Data, Panda) but I suppose they are not as effective as MBAE or HMPA.

    [EDIT] Panda added in last paragraph.
     
    Last edited by a moderator: Apr 11, 2015
  10. 142395

    142395 Guest

    I agree with Rasheed, I feel you guys expect too much for the test idealistically, as if this is a comparative tests. The 2 major purpose a vendoer apply sponsored test is, 1) see how the product perform 2) to find problem in the product. If results was good, they may publish it to promote sales. If bad, they can choose not to publish it legally. And they can use feedback to improve their products, and MRG giving feedback is not the first time, Zemana got feedback and used it to improve their product. I know, your arugument is they re-tested after feedback. But remember, Surfright could choose not to publish first test and only publish 2nd test. If they did, we can't see those thing, I think it's much worse. But they didn't, because they don't need to hide it. I noticed it when I looked it, but didn't mentioned it cuz it's not matter. 2 versions of HMPA is clearly in the graph.

    What required in tests are 2 thing, 1) don't do injustice like data manipulation 2) be transparent. We can't know if they test justly, and as to transparency every testing organization are not enough, AVC is best within it, but this MRG test is IMO relatively transparent.
     
  11. Minimalist

    Minimalist Registered Member

    Joined:
    Jan 6, 2014
    Posts:
    13,853
    Location:
    Slovenia
    I agree. If they decide to use this results for marketing purposes, customers will see that there were two versions tested and that they improved their defenses after they got results from earlier version. IMO it would be unethical if they would hide this fact, but since they've published both results, I don't see much harm.
     
  12. ropchain

    ropchain Registered Member

    Joined:
    Mar 26, 2015
    Posts:
    335
    I am currently preparing a demo of a HMPA bypass as a reaction to the report, stay tuned...
     
  13. Gapliin

    Gapliin Registered Member

    Joined:
    Feb 12, 2012
    Posts:
    81
    Responsible and not full disclosure, I hope?
     
  14. ropchain

    ropchain Registered Member

    Joined:
    Mar 26, 2015
    Posts:
    335
    Not full disclosure, just a screen capture.
    (It also impacts EMET and MBAE)
     
  15. Rasheed187

    Rasheed187 Registered Member

    Joined:
    Jul 10, 2004
    Posts:
    13,412
    Location:
    The Netherlands
    Yes, I was mainly talking about "application lockdown" which it lacks with standard configuration I suppose. I don't really like EMET, it looks a bit complex.
     
  16. ropchain

    ropchain Registered Member

    Joined:
    Mar 26, 2015
    Posts:
    335
    I have experienced EMET as being too slow when EAF+ is enabled. Without EAF+ EMET is not causing any major slowdowns. Although EAF+ is a key feature against info leaks.
     
  17. Rasheed187

    Rasheed187 Registered Member

    Joined:
    Jul 10, 2004
    Posts:
    13,412
    Location:
    The Netherlands
    What I basically meant, was that I wondered if the payloads were blocked (from running) via anti-exploit techniques (HIPS) or with signature. What happens after execution is not relevant since this is no behavior blocking or containment test. But if AV's were not given a chance to block the payload with signature then it would be unfair indeed. I'm not sure if this was the case.
     
  18. Rasheed187

    Rasheed187 Registered Member

    Joined:
    Jul 10, 2004
    Posts:
    13,412
    Location:
    The Netherlands
    I don't think it was done with the intention to make them look bad. At the end of the day, AV's claim to be able to stop malware, it shouldn't matter how it's delivered. You can clearly see that AV's like Norton and Kaspersky perform quite well, it's because they have a special anti-exploit module. ESET was a bit disappointing since they have also been offering anti-exploit for a couple of years now. Bitdefender and Avast performed nicely. The ones that scored under 80% should up their game.
     
  19. Rasheed187

    Rasheed187 Registered Member

    Joined:
    Jul 10, 2004
    Posts:
    13,412
    Location:
    The Netherlands
    Exactly my point, MRG have been quite transparent, so I don't understand all the fuzz. I don't see what's so wrong about the "artificial test", it's simply a showcase for a feature that other apps lack, big deal. It wasn't even included in the main exploit blocking test.
     
  20. erikloman

    erikloman Developer

    Joined:
    Jun 4, 2009
    Posts:
    3,152
    Location:
    Hengelo, The Netherlands
    A tool like ProcessMonitor was used to see wether the malware process was actually started.
     
  21. Rasheed187

    Rasheed187 Registered Member

    Joined:
    Jul 10, 2004
    Posts:
    13,412
    Location:
    The Netherlands
    OK, so I suppose that if the malware process was loaded and active in memory, without an AV (or anti-exploit tool) alerting about it, it was considered a fail? Sounds logical to me. Because this also means that if HIPS/anti-exploit was bypassed, the AV was apparently not able to stop it with a signature as well.
     
  22. Peter2150

    Peter2150 Global Moderator

    Joined:
    Sep 20, 2003
    Posts:
    20,590
    I have testet several real crypt stuff, and indeed HMPA shut it down. BUT.. before they could get to HMPA EMsisoft did block them, so I consider it an excellent test of HMPA but as a comparison test it was totally bogus in my mind
     
  23. Gapliin

    Gapliin Registered Member

    Joined:
    Feb 12, 2012
    Posts:
    81
    They definitely were able to use signatures or blacklisting. According to the text the following levels were counted as success for the products:
    So actually simply blocking the URL would be enough. Which imo invalidates the test if they were really trying to test whether a product does detect (and prevent) exploits or not.
    They even say this themselves:
    It's not about how it's delivered, that's not the point. Anti-malware suites claim they will be able to stop malware, but with all enabled modules. That's why most of them use different layers.
    I, for one, am only complaining about chapter 4, the "product comparism". Claiming to test HMPA "alongside twelve competitor products" even though some are no competitors is not fair. Some products simply have no exploit blocking abilities, so why including them?.
    In fact I am completely fine with all the other stuff. The product review of HMPA is great. Making all the effort to actually develop a simple Firefox plugin and then exploit it, adding all the source-code is really impressive and I applaud them for their transparency and - mostly - for their methodology (especially if you compare it to the test by PCSL).
    Nope. It was considered a fail if a new process (the payload) spawned. That's it. Any alerts afterwards were discarded.
     
  24. itman

    itman Registered Member

    Joined:
    Jun 22, 2010
    Posts:
    8,111
    Location:
    U.S.A.
    Hum ......... I have used EAF + to monitor non-browser app .dlls with no performance hit whatsoever. Also I use IE and have had no performance impact while browsing.
     
  25. CloneRanger

    CloneRanger Registered Member

    Joined:
    Jan 4, 2006
    Posts:
    4,979
    No tests are perfect, but i've always liked what MRG has done. As it's helped make HMPA even better, that's a good thing.
     
  1. This site uses cookies to help personalise content, tailor your experience and to keep you logged in if you register.
    By continuing to use this site, you are consenting to our use of cookies.