MRG Effitas – Real World Exploit Prevention – March 2015 (sponsored by Surfright)

Discussion in 'other anti-malware software' started by FleischmannTV, Apr 7, 2015.

  1. FleischmannTV

    FleischmannTV Registered Member

    Joined:
    Apr 7, 2013
    Posts:
    1,071
    Location:
    Germany
    Project details: MRG Effitas Real World Exploit Prevention Test March 2015

    • Diverse set of exploit kits (12)
    • Diverse set of vulnerabilities (16 different CVEs) in the product comparison
    • Internet Explorer, Firefox and Chrome exploits used
    • Large number of internet security suites and anti-exploit tools – 13 products
    • Use of in-the-wild in-memory malware
    • Test with an artificial zero-day attack
    • Manual test and result analysis
    • Combined in-the-wild and Metasploit test
    • Sponsored by SurfRight – HitmanPro Alertv3


    Source:

    https://www.mrg-effitas.com/mrg-effitas-real-world-exploit-prevention-test-march-2015/
     
  2. Mayahana

    Mayahana Banned

    Joined:
    Sep 13, 2014
    Posts:
    2,220
    Well look at that.. Norton 2015 right up near the top, almost as good as HMPA itself. Trend fails at this, which is why they are beta testing their exploit prevention mechanics right now, with a potential for Trend 2016 to include an advanced anti-exploit engine.

    Norton itself, I love their IPS, Exploit, and Insight protection systems. VERY advanced! I think Norton 2015 is one of my favorite products for this year, it's consistently a good performer. What will next year bring? How will the Trend+Norton battle heat up with their 2016 product lineups?
     
  3. Minimalist

    Minimalist Registered Member

    Joined:
    Jan 6, 2014
    Posts:
    5,061
    Interesting results, thanks for sharing.
    I thought that anti-exploit tools would have huge advantage over anti-malware software, but that's not case in this test. I will surely leave exploit blocker enabled on my ESET installation.
     
  4. erikloman

    erikloman Developer

    Joined:
    Jun 4, 2009
    Posts:
    3,029
    Location:
    Hengelo, The Netherlands
    Do not be fooled by the results of the anti-malware software. They have plenty of time to block the old exploits used in the test. Its the meterpreter tests that somewhat reflect the effectiveness against zero-days.

    From page 33:

    Note: Because it is difficult to obtain a diverse and large number of fresh in-the-wild exploits for this comparison, at the time of testing the in-the-wild exploit landing pages were already known for a considerable period of time. For security suites like Norton, Avast, Kaspersky and Bitdefender, it is reasonably expected that they will block these pages and payloads using reactive blacklist-based technologies that rely on prior discovery, like URL filtering, virus signatures.
     
  5. Minimalist

    Minimalist Registered Member

    Joined:
    Jan 6, 2014
    Posts:
    5,061
    Yes, too bad that they didn't show how specific exploit was blocked (pro-actively or by signature). That would probably be more complicated to achieve as AM software blocks exploits by using different approaches. So in the end all we have is that chart and blocked/fail for specific exploit.

    EDIT: I also use ESET with some protections disabled (protocol filtering), so I can't tell how good ESET would be against this exploits with only anti-exploit module enabled...
     
  6. Nightwalker

    Nightwalker Registered Member

    Joined:
    Nov 7, 2008
    Posts:
    784
    Amazing results from Hitman Pro Alert, it is a very solid product for sure :thumb:
     
  7. FleischmannTV

    FleischmannTV Registered Member

    Joined:
    Apr 7, 2013
    Posts:
    1,071
    Location:
    Germany
    This test was needed after the PCSL low blow "test" from August 2014.

    Congratulations to Surfright and the Loman brothers, well deserved.
     
  8. lordraiden

    lordraiden Registered Member

    Joined:
    Jan 30, 2006
    Posts:
    3,067
    Is there a difference or the free version should have obtained the same results?

    Why Webroot secure anywhere hasn't been tested? is probably one of the few AV that could have done something against exploits.
     
    Last edited: Apr 7, 2015
  9. Mayahana

    Mayahana Banned

    Joined:
    Sep 13, 2014
    Posts:
    2,220
    Haven't Norton has always been strong against exploits, but that much stronger with the IPS integration from SEP into the consumer product - which seems to be paying off. Very impressive results.

    Also curious as to why Webroot wasn't on deck for this one?
     
  10. J_L

    J_L Registered Member

    Joined:
    Nov 6, 2009
    Posts:
    8,516
    Avast doing quite well yet again. Colour me surprised, as I have the free version and HMP.A 3.
     
  11. itman

    itman Registered Member

    Joined:
    Jun 22, 2010
    Posts:
    2,969
    Location:
    U.S.A.
    Interesting comment in the study:
    Configuring EMET to protect Flash on Firefox is mission impossible, due to the different filenames of Flash for every version.

    Confirms my assumptions that EMET works best with IE. Also note that EMET was tested using it's default configuration.

    I am surprised at Eset's results considering they did add exploit and memory protection to Smart Security 8.

    And Emsisoft better get its act together on the exploit issue. To be beat out by MSE, oh my ...............
     
  12. BoerenkoolMetWorst

    BoerenkoolMetWorst Registered Member

    Joined:
    Dec 22, 2009
    Posts:
    3,766
    Location:
    Outer space
    You indeed have to re-add it after every update of Flash. Unfortunately, EMET allows wildcards only on file paths not filenames.
    Here's a way to make it easier though: https://www.wilderssecurity.com/thr...xperience-toolkit.344631/page-26#post-2406419

    The fact that EMET's popular software profile default rules make Firefox's boot time a lot longer, and the fact that it's only included in the popular and not recommended profile, would indeed suggest that it receives a lot less priority than IE.
     
  13. FleischmannTV

    FleischmannTV Registered Member

    Joined:
    Apr 7, 2013
    Posts:
    1,071
    Location:
    Germany
    I have a suspicion. My guess is Webroot is one of their few paying customers. They probably would have sucked big time at the exploit prevention part and there was neither any marketing value in this test for Webroot nor would it have helped MRG Effitas to keep them as a customer. So neither party had any motivation for them to participate here.

    In my opinion this is one of the few remaining AVs that do the least against exploits, be it reactively or proactively.

    It's kind of strange that Webroot isn't included, even though they are a customer, whereas Emsisoft is included, despite the termination of the business relationship. So as a customer you can be exluded from testing, as a non-customer you cannot. Shamed be he who thinks evil of it. :isay:
     
  14. Windows_Security

    Windows_Security Registered Member

    Joined:
    Mar 2, 2013
    Posts:
    3,079
    Location:
    Netherlands
    @FleischmannTV Good explanation, there are more clues which lead to this suspicion.

    This test was crafted especially for HPMA 3. Pitty the artificial test crafted by MRG did show that HPMA was best, so they decided to :isay:

    Well done Surfright & MRG Effitas, congratulations with this result. :argh:
     
    Last edited: Apr 7, 2015
  15. Nightwalker

    Nightwalker Registered Member

    Joined:
    Nov 7, 2008
    Posts:
    784
    The first bad result of Emsisoft in MRG testing, coincidences happens I guess :ninja:
     
  16. Fabian Wosar

    Fabian Wosar Developer

    Joined:
    Aug 26, 2010
    Posts:
    787
    Location:
    Germany
    I stopped reading after this:
    No idea who's brilliant idea it was to include products that don't even advertise or feature exploit protection and then design the test so that all other additional layers that would detect payloads, like for example behavior blocking, will be completely ignored :isay:.
     
  17. Peter2150

    Peter2150 Global Moderator

    Joined:
    Sep 20, 2003
    Posts:
    17,043
    While not taking a thing from HMPA, because I think it a great product, I also totally agree with Fabian. I thrown a lot of things at HMPA and it has always worked....BUT...I've always had to disable EIS first, or it blocked them.

    This could have been a good test to show that HMPA did indeed work, but the want one step to far and took the test totally down hill
     
  18. Cutting_Edgetech

    Cutting_Edgetech Registered Member

    Joined:
    Mar 30, 2006
    Posts:
    4,950
    Location:
    USA
    I've only had a chance to glance over parts of the test, but this test does not really measure a traditional black listing Antiviruses ability to block exploits in the browser. If the AV blocked the link, or IP of the page containing the exploit it was counted as blocking the exploit. An AV with really aggressive blacklisting of links, and IP's could possibly receive a 100% score without being able to block a single exploit on any of the webpages. Products like HMPA, MBAE, and EMET which do not use blacklisting of links, or IP's have to actually block the exploits in order to receive a block. I don't think the test reveals the method in which each exploit was blocked by the AV so what we can take from this test is that X AV does a good/bad job in mitigating exploits using whatever methods they choose to use. I have only glanced over parts of the test so far. If they used exploits for office documents, adobe files, etc. outside the browser then the results could be drastically different. The AV could avoid blocking the actual exploit though by blacklisting the files so the user never has the change to execute the files containing the exploits. There are so many variables when testing with the protocol they used. I will look through the test procedures a little later to see what other type of test they conducted, and in what environment.
     
  19. Frank the Perv

    Frank the Perv Banned

    Joined:
    Dec 16, 2005
    Posts:
    882
    Location:
    Virginia, USA
    Normally test-bashers are self-serving and annoying. All tests have constraints, assumptions, and limitations.

    But lots of legit critical commentary on this one.

    I dunno about this one..
     
  20. cruelsister

    cruelsister Registered Member

    Joined:
    Nov 6, 2007
    Posts:
    976
    Location:
    Paris
    There was a movie made many years ago called Major League. In one scene two coaches were watching a recently called up from the Minors player consistently hit pitches out of the park during batting practice. One coach asked the other why this player wasn't in the major leagues earlier. The response was "Throw him a curve" (which he didn't come close to hitting).

    I suggest the authors of this study should throw HMP a worm. The results wouldn't be so pretty.
     
  21. Yuki2718

    Yuki2718 Registered Member

    Joined:
    Aug 15, 2014
    Posts:
    1,257
    Complete nonsense. HMPA is not designed to stop malware, tho some components can stop certain malware action.
    The feature tested is designed to stop remote code-execution exploit which abuses (mainly memory corruption) application vulnerability.

    I really don't get see some ppl want everything in a program especially complementary program like HMPA, and this sentiment is not limited to HMPA, I feel this often.
     
  22. Yuki2718

    Yuki2718 Registered Member

    Joined:
    Aug 15, 2014
    Posts:
    1,257
    As already noted, it's not pure exploit test nor real-world test as it doesn't include behavior blocking. I wonder why they used such half-baked scheme. In earlier corporate test they did real-world test and separate results to pure-exploit and dynamic protection, which I think is much better. Now in this new scheme we can't tell how each product is good in exploit prevention nor whole dynamic prevention.

    That said, still we can get some insight from this half-baked test.
    All high scored AVs (Norton, Kaspersky, Avast) have strong NIPS. Yeah, NIPS is signature based detection so is not capable to block 0day, however it doesn't mean NIPS is useless. NIPS shorten time gap btwn new vuln discovery and official patch. Some vuln can't be prevented by anti-exploit but can be by NIPS (kernel exploit, logic-flaw).

    As to simulator test, it makes sense as it would be only way to test unknown 0day exploit, but sure, their "provided" test won't be persuasive in such forum where there're many knowledgeable and critical ppl, even when the exploit is publicly available and criminal may use it or similar. I don't doubt Malwarebytes can make PoC in which only MBAE can block.

    Honestly, tho it's great HMPA can block ITW exploit, what I want to see is not this. I'd like to see continuous challenge against HMPA by skilled researcher. EMET have been challenged and improved much. MBAE also have been challenged tho doesn't reach EMET's degree. IMO, it is MUST-have for HMPA to be really bullet-proof product. Also, I want to know if all exploits in this test were blocked by HMPA's memory protection or application lockdown but not process protection.

    @Mayahana
    Trend would have earned better score if they enabled firewall tuner (IIRC, it's default-off.) which is basically NIPS. Same goes to nearly all exploit tests including MRG's corporate exploit test (again, NIPS is not included in default OfficeScan, it's an addon) except ones performed by NSS Labs where they allowed each vendor configure their product.

    @lordraiden
    I'm sure Webroot wouldn't get good score as they don't have ANY NIPS component even in premium suite. It just means their protection philosophy is different, they're more focusing on malware blocking, same as Emsi.

    @FleischmannTV, @Nightwalker I would feel the same if it was full real-world test, but as Fabian said it's not and Emsi currently don't have strong NIPS nor behavior-based anti-exploit, so I don't feel need for conspiracy theory to explain the result.
    Generally, most "suspicion" can be reasonably explained if one have enough knowledge about technically how each product works and testing methodology. The problem is ppl tend to think good products are good in all aspect (there's no such product), and often speak before they read methodology.
     
  23. Tarnak

    Tarnak Registered Member

    Joined:
    Feb 5, 2007
    Posts:
    3,875
    I don't go by tests...I invest my time in a product, and beta test. Very simple!
     
  24. markloman

    markloman Developer

    Joined:
    Jan 25, 2005
    Posts:
    433
    Location:
    Hengelo
    Actually, the ROP chain and shellcode that we provided for the artificial zero-day attack defeats our HitmanPro.Alert 3 as well, when you run the attack inside a virtual environment or on non-Intel hardware. We went all the way to beat anti-exploit solutions, including our own.
     
  25. Tarnak

    Tarnak Registered Member

    Joined:
    Feb 5, 2007
    Posts:
    3,875
    So, how do I test that claim in my own system that is running HMP.A... I don't have a VM installed, and I don't want to. ;)