MRG Effitas – Real World Exploit Prevention – March 2015 (sponsored by Surfright)

Discussion in 'other anti-malware software' started by FleischmannTV, Apr 7, 2015.

  1. Rasheed187

    Rasheed187 Registered Member

    Joined:
    Jul 10, 2004
    Posts:
    13,435
    Location:
    The Netherlands
    This actually validates the test, because while tools like HMPA and MBAE could only rely on pro active detection, AV's even had a chance to pass the test by blocking them with other modules. That's the job of AV's, to stop malware no matter if it's triggered by exploit or by user installation. So they didn't even have to feature a dedicated exploit blocker. And this test was about stopping the payload from running, not about blocking suspicious behavior triggered by the payload.
     
  2. Rasheed187

    Rasheed187 Registered Member

    Joined:
    Jul 10, 2004
    Posts:
    13,435
    Location:
    The Netherlands
    I don't see how that translates into a bogus test. You had a couple of good experiences with EIS, and of course it's probably a quite powerful tool against malware, but I think test was more extensive. And besides, other AV's like Norton did perform well.
     
  3. Peter2150

    Peter2150 Global Moderator

    Joined:
    Sep 20, 2003
    Posts:
    20,590
    I think comparing Non exploit software against exploit software isn't a valid test.
     
  4. Gapliin

    Gapliin Registered Member

    Joined:
    Feb 12, 2012
    Posts:
    81
    That's ultimately what I'm complaining about. The methodology. If you simply compare all different sorts of security suites it's not fair imo to alter the rules so you actually look like the definite winner. Yea, I know that's how it goes...

    An example: Add some anti-executable software to the mix. Boom - 100% blocked (at least for file-based payloads). Since it's based on whitelisting. But they didn't because it would be obvious that you're comparing apples and oranges.
    It's all about the viewpoint and about the explanation why you're comparing those products and what you're trying to achieve with it.

    Anyways... we could go on for hours...
     
  5. ropchain

    ropchain Registered Member

    Joined:
    Mar 26, 2015
    Posts:
    335
    First of all: Sorry for my bad English.

    NB: This post is not meant to bash on any tool, person or company, just to indicate that bypassing HMP.Alert is still possible despite of a 100% coverage of build 174 in the “Real World Exploit Prevention Test” report written by MRG Effitas. Please note that the existence of a bypass does not mean that a mitigation tool does not work. Earlier reports and demonstration videos have already shown that HitmanPro.Alert and Malwarebytes Anti-Exploit are able to block almost all of the In-the-wild deployed exploit code. Creating a basic proof of concept bypass still takes some time, but it should not be impossible for a motivated researcher.

    The following demonstration shows a Windows 7 machine running an outdated version of Internet Explorer 8 strengthened with HitmanPro.Alert 3 build 181. Before running the actual proof-of-concept bypass I will show that HitmanPro.Alert is actually protecting Internet Explorer against attacks.

    ~Link Removed. See this. https://www.wilderssecurity.com/threads/posting-policy-recommended-threads.180128/#post-1041384~

    Just a few notes about this proof-of-concept bypass:
    - Bypassing Application Lockdown is beyond the scope of this demo.
    - The same technique also affects MBAE and EMET.

    Although I have to say that the effectiveness of HMP.Alert has been improved quite a lot in the latest builds.
     
    Last edited by a moderator: Apr 10, 2015
  6. TonyW

    TonyW Registered Member

    Joined:
    Oct 12, 2005
    Posts:
    2,741
    Location:
    UK
    It is to be hoped then that the researcher provides information to the relevant vendors of the PoC bypass so that their product(s) can be improved upon for the benefit of those who use them.
     
  7. Mayahana

    Mayahana Banned

    Joined:
    Sep 13, 2014
    Posts:
    2,220
    Agree - 100%.. It's like tossing a Yugo into a Nascar race to prove how fast your Nascar rig is.. Invalid!
     
  8. 142395

    142395 Guest

    I rather wish MRG discarded URL blocking part as that is no relevant to exploit, maybe by transferring the URL to other virtual server, or by allowing site access in AV's UI. Then it will come to pure exploit test.

    But I don't see your argument that those AVs which don't have anti-exploit shouldn't be included. I want to know how each product performs against exploit. I don't use AV alone, always combine with other products. If your AV lacks anti-exploit capability (or have it but weak), you'll want to add anti-exploit. If lacks strong behavior protection, stand alone HIPS or so. Those specific test is very useful for those who understand each AV's technology and read methodology, and can take correct context. It can't be comparable to whole dynamic test and for that we already have many tests.

    And I have some suspicion that, do all of you know what AV have anti-exploit capability (either NIPS, behavior, or other) and what not? It might be possible somebody wrongly assume his AV can (or can't) block exploit when the truth is opposite. Exploit itself is already dangerous, so stopping it in earlier stage have a value. Not all exploit download malware. Like static file detection test still have some value, pure exploit test have value. Only problem is they did half-baked test which is not pure exploit nor whole dynamic, and they didn't classified the results.
     
  9. FleischmannTV

    FleischmannTV Registered Member

    Joined:
    Apr 7, 2013
    Posts:
    1,094
    Location:
    Germany
    Because a lot of people don't look at the methodology. Immediately after the test I was able to witness reactions across different forums where people were asking what the hell was going on with Emsisoft and that they needed to get their act together. Then Fabian Wosar explained it to them and suddenly they were relieved. Most people simply look at the numbers and that's it.
     
    Last edited: Apr 11, 2015
  10. aztony

    aztony Registered Member

    Joined:
    Sep 9, 2012
    Posts:
    595
    Location:
    Phoenix, AZ
    Very true!
     
  11. Rasheed187

    Rasheed187 Registered Member

    Joined:
    Jul 10, 2004
    Posts:
    13,435
    Location:
    The Netherlands
    Well, perhaps I may be misunderstanding, so let's get to the bottom of it. Are you saying that for example Emsisoft would have most likely blocked the malware (delivered via exploit) with the real-time scanner, after the malware is already active in memory? If so, then I agree it's a bit unfair to AV's, because then it wouldn't show the whole picture.

    But the way I understand it, is that Emsisoft and the others who did badly, simply could not identify or block the malware at all, even when they were already running in memory. And that would be a fail indeed. Also, I'm sure that EIS's behavior blocker would probably alert about (or block) the suspicious behavior from the malware, but that is out of the scope of this test.
     
  12. Rasheed187

    Rasheed187 Registered Member

    Joined:
    Jul 10, 2004
    Posts:
    13,435
    Location:
    The Netherlands
    I disagree, AV's should be able to block exploits. There are several ways you can block exploits and payloads. You don't have to use the advanced techniques used by HMPA, MBAE and EMET.

    Yes, anti-executable would probably perform well, but the reason why they are not included is because they simply lock down the whole system, they block both legitimate and malicious software.
     
  13. Peter2150

    Peter2150 Global Moderator

    Joined:
    Sep 20, 2003
    Posts:
    20,590
    Hi Rasheed

    I've tested against the crypto garbage that comes via email, and yes EIS first detects it as a trojan, and if I turn off File guard, which knocks out the AV, and then yes the BB shuts it down. So bottom line is EIS does protect, but not from the exploit but the payload. Same way although not tested Appguard won't stop the exploit but it would have protected the system, so the tests while valid, are also a bit misleading.
     
  14. Rasheed187

    Rasheed187 Registered Member

    Joined:
    Jul 10, 2004
    Posts:
    13,435
    Location:
    The Netherlands
    Yes, and that's what I'm trying to figure out. So basically this test was only about stopping exploits, and not about blocking payloads? I hope SurfRight can clarify this, because if you can block the payload from doing any damage by terminating them immediately after they become active in memory, then you have basically also passed the test. And if so, I agree that only AV's with a dedicated exploit blocker (like Norton, Kaspersky and ESET) should have been tested.
     
  15. itman

    itman Registered Member

    Joined:
    Jun 22, 2010
    Posts:
    8,120
    Location:
    U.S.A.
    Last fall, China based PCSL another uncertified security lab was commissioned by MalwareBytes to test MBAE against exploits.

    http://pcsl.r.worldssl.net/report/exploit/rce_mitigations_201408_en_malwarebytes.pdf

    Not too surprisingly, MBAE scored no. 1 in the testing. Additionally in that test, EMET ranked third highest; right behind Norton IS. Also, HMPA didn't so so well in that test. It is given that there has been significant enhancements to HPMA since this test was performed.

    Is it just a bit not coincidental that the sponsoring i.e. the one paying for the test is always the one that comes out on top of these "independent" tests? Personally, I believe these tests are nothing more than a modern spin on the old magician's "smoke and mirrors" trick. I am sticking with EMET for my exploit protection. It is not perfect as we all know no security solution is. But it is free and with EAM paid, I feel I am more than adequately protected.
     
  16. 142395

    142395 Guest

    Rasheed, before requiring clarification you have to read the methodology.
    If a product stopped the payload, it passes the test (4.1 p31).
     
  17. Rasheed187

    Rasheed187 Registered Member

    Joined:
    Jul 10, 2004
    Posts:
    13,435
    Location:
    The Netherlands
    Well, that's the thing, how to interpret the results? I wonder how bad performers like Emsisoft and Trend Micro did manage to block some payloads, that would clear things up a bit. Were they allowed to use HIPS/BB? And if so, could HIPS have stopped and terminated the payloads? Because when the payload was loaded, it was already a fail.

    And let's take EXE Radar Pro (ERP) as example, it actually lets all new processes execute in sort of a "suspended state", and if it's not on the white-list, it's immediately killed. So would ERP have scored 0% in this test, even though the payload couldn't do any damage? These are important things to consider, before we can decide if it was a reasonably fair test or not.
     
    Last edited: Apr 12, 2015
  18. Sampei Nihira

    Sampei Nihira Registered Member

    Joined:
    Apr 7, 2013
    Posts:
    1,881
    Location:
    Italy
    @Zoltan_MRG

    You can know the mitigation intervened in the following tests?

    MBAE = 005/006/007/012/014/033/037/039.

    EMET= 002/022/023/028
     
  19. Zoltan_MRG

    Zoltan_MRG Registered Member

    Joined:
    Apr 9, 2015
    Posts:
    31
    Please excuse me for the late reply, I was on holiday. We value your feedback, so here is our response.

    I don’t see any reference on page 43 about Chrome exploits. Please don’t get confused about the string chrome:// in the report. This was a Firefox exploit. The in-the-wild product comparison test 006 was targeting Silverlight in Chrome, test 005, 007, 012 and 014 targeting Flash in Firefox.

    Also, in post #58 “some of the protections haveno existing real world exploits yet for Chrome.” I don’t understand this sentence, “protections .. have … exploits”?

    And as answered in post #72, “1. Determining offsets can all be done dynamically, as long as you have RW access to memory on a 32 bit browser.”

    Default settings are highly important during the test. These products are developed for home users, and home users don’t like to fine-tune their product. If a product wants better results, turn on all protections by default … But if this makes problems (e.g. false-positives), then the vendor or the user will turn it off – which means the feature is not that good…

    All of our tests were done via the following way: At the beginning of the test, we googled for the most recent version of the product, and from the official site, we downloaded the latest version. After installing, we checked all update has been installed, the OS is restarted. And before every test, we allowed the auto-update to finish. And yes, it is possible that some newer builds were available at the time of testing, but we were using the same version as average users would. MBAE 1.0.6 was not available at the time of the test for average users. HMPA 3.0.34.174 was available for the home users when we tested it.

    Some test labs don’t allow this, others allow this. I think all of us want’s to live in a better world, and when we help vendors to improve their products, the users will benefit from it. And because we are transparent, we documented all of this in our report.

    We use this methodology for a long time. We don’t consider behavior blocking a success, because it is damage control, not a protection. Adding behavior blocking to the test creates a lot of problems, which is not dealt with in other tests. Here are some problems:

    1. No one knows what happened before the behavior protection blocked the malware. It is possible it already stole all stored passwords and sent it to the C&C server.
    2. It is possible the malware dropped another malware which will activate only after reboot. Or it created file-less persistent malware, like Poweliks.
    3. It is possible the malware made DLL injection or thread injection, thus monitoring running processes is not enough.
    4. It is possible the malware won’t do anything in 1 hour, thus behavior protection can only kick in after 1 hour, but the test will be terminated until that time.
    5. It is possible the behavior protection only blocks the communication with the C&C. If this is a crypto-ransomware, this makes things worse.
    6. It is possible the malware terminated itself because it found traces of a test system, or virtualization. Might be the AV terminated it, might be the malware itself.
    7. In the case of in-memory-malware like Bedep, it is possible the AV blocks future dropped malware, but not the Bedep in-memory, which can still perform it’s malicious actions.

    Checking all of these makes the whole test too complicated, too time-consuming. Especially, because checking/tracing all of this can change the malware behavior (e.g. debugger detection).

    If you have any statistics about how many home users have the hardware valid for this test, I would appreciate that. My bet is on that more than two-third of home users have this kind of protection, or even more.

    On the other hand, because we were transparent and honest, we wrote in the report that this attack was delivered by the vendor. We could have lied about this, but we did not. And because we agree that this test might be seen as not-fair, we openly published everything about the test, so others can validate our results. I have not seen any comment about any technical issues in this test, or why this is invalid. As all details have been published, all competitors are welcome to improve their product. Every detail about this test was given to everyone for free.

    This test demonstrates one thing only: Software based protection can be bypassed in a GENERIC way and hardware assisted protection can block this kind of generic attack. Nothing more, nothing less.

    Let us know where our test failed these ethics guidelines.

    Some other comments we would like to address:

    1. Why was URL blocking not disabled during the test?

    Home users are not interested in metrics like this. Although we believe it will be an interesting test, the results would be hard to interpret.​

    2. Why was product A included in the test, and product B excluded?

    We and the vendor created the list of products to be tested based on the market share, and the relevance. By relevance, we mean either Internet Security Suites or Exploit mitigation tools. As not many home users use Exploit Mitigation tools, it is important to raise awareness among the users that traditional Internet Security Suites without any dedicated exploit protection has worse protection than those which have these protections. And people buy Internet Security Suites to protect their system – this is what they expect. Home users have no idea about the difference of Internet Security Suites and Exploit mitigation programs. It’s like the old days of antivirus protecting against viruses and antispyware protecting against spyware. Both stuff is bad, and should be blocked. Exploits are bad; they should be blocked – no matter how the industry labels the product. For example the prevalent Angler exploit kit with the in-memory Bedep malware bypassed most traditional Internet Security Suites – because the malware delivery won’t trigger most checkpoints in a traditional Internet Security Suite, as the malware can be blocked via only URL blocking (reactive technology) or HTML/Flash analysis (reactive technology). We recommend all the vendors who don’t have exploit protection to implement it, as it will increase the security of their product significantly.​

    3. This is a synthetic test, not a real world one.

    I agree on this, but all tests are synthetic. Some notes why the world is not black and white:

    a. Most “real world tests” are done with significant delays (days, weeks, months?) between the time of recording the exploit and the time of test.

    b. We have seen Metasploit exploits in-the-wild. The “Rocket kitten” campaign was done via Core Impact. In our tests, our setup was done with new, unknown URL’s, high obfuscation level, and in-memory-malware. This is what the bad guys do on an everyday basis. This test simulates the first victims during the day, before new signatures and black lists are shipped (or known in the cloud). In some aspects, Metasploit tests are closer to real world tests than exploits replayed days (or weeks) later.

    c. The last test (the artificial zero day) is indeed a synthetic test. But as we saw it this year, (Flash) zero days happen, even in exploit kits. Any product with exploit mitigation could block it.​

    4. Sponsored tests are <insert any comment here>

    Product comparison tests are like the Olympic games. Sponsored product comparison tests are like the Olympic games on the home field, where the vendor can choose the time when the games will happen, while the competitors have no idea about it. No wonder the vendor who sponsors the test will organize the test when it is stronger than the competition and in a game where the vendor is the best. But this still means the vendor is the best at a given time, in a given game.​

    5. Did you correct Office 20010 bug in past report which I posted in a past MRG test thread?

    Thank you for reporting the error, we saw your comment.
    6. Last fall, China based PCSL another uncertified security lab was commissioned by MalwareBytes to test MBAE against exploits. (Post #115)

    Your post implies that there are “certified” and “uncertified” security labs, this is not the case.

    I think perhaps what you are really trying to say is that some labs are members of AMTSO and others are not. For the avoidance of doubt, MRG Effitas is a member of AMTSO.

    If you take the time to look here - http://www.amtso.org/members you will see PCSL Security Labs is also a member.

    It could be a case that you mean that tests are conducted by individuals who do not have relevant IT certifications. If this is the case, have a look at my Linkedin page for my certificates: https://nl.linkedin.com/in/zbalazs

    If I am misinterpreting your statement, you have my apologies and I would welcome your explanation of what you mean by “certified” or “uncertified” security lab.​
     
  20. itman

    itman Registered Member

    Joined:
    Jun 22, 2010
    Posts:
    8,120
    Location:
    U.S.A.
    Membership in AMTSO does not mean certification. That org is worthless in my opinion. I was referring to ISO certifications.

    ref: https://www.icsalabs.com/accreditationshttps://www.icsalabs.com/accreditations
     
    Last edited: Apr 13, 2015
  21. ropchain

    ropchain Registered Member

    Joined:
    Mar 26, 2015
    Posts:
    335
    Well, I am pretty sure that no test organisation would falsify results in favor of one specific vendor. Looking at the results of HMPA and MBAE I am pretty sure that they are representing the truth.

    But being critical is always a good thing.
     
  22. Zoltan_MRG

    Zoltan_MRG Registered Member

    Joined:
    Apr 9, 2015
    Posts:
    31
    An ISO certification would add no value to our organisation or to the quality of services provided to our clients.
    ISO certifications certify a way a company works, its processes etc, but generally has nothing to do about the technical details. It would and could not cater for the minutiae required when we design a new methodology or conduct unique research for a client as we work in an environment that is so diverse and changing so quickly.
    Also, an ISO certification simply confirms that an organisation does in fact operate in the way it purports to. It does not certify that a certified organisations operations or methodologies etc are necessarily fit for purpose.
    Ultimately, we are talking about the quality; transparency and rigour of services aren’t we. Since this thread is about work we have conducted for SurfRight, I suggest you ask them what they think about the quality of our services, if they were appropriate, did they meet expectations and if they think they have benefitted by engaging us.
    Please understand, this must be my final response to this specific query because as I’m sure you understand I and all the people in the Effitas Group are very busy.
    I am of course willing to answer any relevant questions concerning other subjects relating to this report.
     
  23. Gapliin

    Gapliin Registered Member

    Joined:
    Feb 12, 2012
    Posts:
    81
     
  24. Frank the Perv

    Frank the Perv Banned

    Joined:
    Dec 16, 2005
    Posts:
    882
    Location:
    Virginia, USA
    Well I hope you had a good holiday.



    I guess most of us knew this, and we know there are other advantages as well. It's refreshing that you state it openly though.


    Thanks,

    -Frank
     
  25. Rasheed187

    Rasheed187 Registered Member

    Joined:
    Jul 10, 2004
    Posts:
    13,435
    Location:
    The Netherlands
    @ Zoltan_MRG

    Thanks for the clarification, I have to say that I agree with them, you make some valid points. What I do would like to know is how tools like Emsisoft and Trend Micro (for example) where able to block some payloads. I'm interested in this, because there was a lot of criticism about the testing of products who didn't have advanced anti-exploit protection. For example, Emsisoft Internet Security has got what they call "Triple layer protection", namely: Surf Protection, Real-time File Guard and Behavior Blocker. So how did they block payloads from running?
     
  1. This site uses cookies to help personalise content, tailor your experience and to keep you logged in if you register.
    By continuing to use this site, you are consenting to our use of cookies.