MRG Effitas – Real World Exploit Prevention – March 2015 (sponsored by Surfright)

Project details: MRG Effitas Real World Exploit Prevention Test March 2015

• Diverse set of exploit kits (12)
• Diverse set of vulnerabilities (16 different CVEs) in the product comparison
• Internet Explorer, Firefox and Chrome exploits used
• Large number of internet security suites and anti-exploit tools – 13 products
• Use of in-the-wild in-memory malware
• Test with an artificial zero-day attack
• Manual test and result analysis
• Combined in-the-wild and Metasploit test
• Sponsored by SurfRight – HitmanPro Alertv3

Source:

https://www.mrg-effitas.com/mrg-effitas-real-world-exploit-prevention-test-march-2015/

 

Well look at that.. Norton 2015 right up near the top, almost as good as HMPA itself. Trend fails at this, which is why they are beta testing their exploit prevention mechanics right now, with a potential for Trend 2016 to include an advanced anti-exploit engine.

Norton itself, I love their IPS, Exploit, and Insight protection systems. VERY advanced! I think Norton 2015 is one of my favorite products for this year, it's consistently a good performer. What will next year bring? How will the Trend+Norton battle heat up with their 2016 product lineups?

 

Interesting results, thanks for sharing.
I thought that anti-exploit tools would have huge advantage over anti-malware software, but that's not case in this test. I will surely leave exploit blocker enabled on my ESET installation.

 

Do not be fooled by the results of the anti-malware software. They have plenty of time to block the old exploits used in the test. Its the meterpreter tests that somewhat reflect the effectiveness against zero-days.

From page 33:

Note: Because it is difficult to obtain a diverse and large number of fresh in-the-wild exploits for this comparison, at the time of testing the in-the-wild exploit landing pages were already known for a considerable period of time. For security suites like Norton, Avast, Kaspersky and Bitdefender, it is reasonably expected that they will block these pages and payloads using reactive blacklist-based technologies that rely on prior discovery, like URL filtering, virus signatures.

 

Yes, too bad that they didn't show how specific exploit was blocked (pro-actively or by signature). That would probably be more complicated to achieve as AM software blocks exploits by using different approaches. So in the end all we have is that chart and blocked/fail for specific exploit.

EDIT: I also use ESET with some protections disabled (protocol filtering), so I can't tell how good ESET would be against this exploits with only anti-exploit module enabled...

 

Amazing results from Hitman Pro Alert, it is a very solid product for sure

 

This test was needed after the PCSL low blow "test" from August 2014.

Congratulations to Surfright and the Loman brothers, well deserved.

 

Is there a difference or the free version should have obtained the same results?

Why Webroot secure anywhere hasn't been tested? is probably one of the few AV that could have done something against exploits.

 

Haven't Norton has always been strong against exploits, but that much stronger with the IPS integration from SEP into the consumer product - which seems to be paying off. Very impressive results.

Also curious as to why Webroot wasn't on deck for this one?

 

Avast doing quite well yet again. Colour me surprised, as I have the free version and HMP.A 3.

 

Interesting comment in the study:
Configuring EMET to protect Flash on Firefox is mission impossible, due to the different filenames of Flash for every version.

Confirms my assumptions that EMET works best with IE. Also note that EMET was tested using it's default configuration.

I am surprised at Eset's results considering they did add exploit and memory protection to Smart Security 8.

And Emsisoft better get its act together on the exploit issue. To be beat out by MSE, oh my ...............

 

You indeed have to re-add it after every update of Flash. Unfortunately, EMET allows wildcards only on file paths not filenames.
Here's a way to make it easier though: https://www.wilderssecurity.com/thr...xperience-toolkit.344631/page-26#post-2406419

The fact that EMET's popular software profile default rules make Firefox's boot time a lot longer, and the fact that it's only included in the popular and not recommended profile, would indeed suggest that it receives a lot less priority than IE.

 

I have a suspicion. My guess is Webroot is one of their few paying customers. They probably would have sucked big time at the exploit prevention part and there was neither any marketing value in this test for Webroot nor would it have helped MRG Effitas to keep them as a customer. So neither party had any motivation for them to participate here.

In my opinion this is one of the few remaining AVs that do the least against exploits, be it reactively or proactively.

It's kind of strange that Webroot isn't included, even though they are a customer, whereas Emsisoft is included, despite the termination of the business relationship. So as a customer you can be exluded from testing, as a non-customer you cannot. Shamed be he who thinks evil of it.

 

@FleischmannTV Good explanation, there are more clues which lead to this suspicion.

This test was crafted especially for HPMA 3. Pitty the artificial test crafted by MRG did show that HPMA was best, so they decided to

Well done Surfright & MRG Effitas, congratulations with this result.

 

The first bad result of Emsisoft in MRG testing, coincidences happens I guess

 

I stopped reading after this:

No idea who's brilliant idea it was to include products that don't even advertise or feature exploit protection and then design the test so that all other additional layers that would detect payloads, like for example behavior blocking, will be completely ignored .

 

I've only had a chance to glance over parts of the test, but this test does not really measure a traditional black listing Antiviruses ability to block exploits in the browser. If the AV blocked the link, or IP of the page containing the exploit it was counted as blocking the exploit. An AV with really aggressive blacklisting of links, and IP's could possibly receive a 100% score without being able to block a single exploit on any of the webpages. Products like HMPA, MBAE, and EMET which do not use blacklisting of links, or IP's have to actually block the exploits in order to receive a block. I don't think the test reveals the method in which each exploit was blocked by the AV so what we can take from this test is that X AV does a good/bad job in mitigating exploits using whatever methods they choose to use. I have only glanced over parts of the test so far. If they used exploits for office documents, adobe files, etc. outside the browser then the results could be drastically different. The AV could avoid blocking the actual exploit though by blacklisting the files so the user never has the change to execute the files containing the exploits. There are so many variables when testing with the protocol they used. I will look through the test procedures a little later to see what other type of test they conducted, and in what environment.

 

Normally test-bashers are self-serving and annoying. All tests have constraints, assumptions, and limitations.

But lots of legit critical commentary on this one.

I dunno about this one..

 

There was a movie made many years ago called Major League. In one scene two coaches were watching a recently called up from the Minors player consistently hit pitches out of the park during batting practice. One coach asked the other why this player wasn't in the major leagues earlier. The response was "Throw him a curve" (which he didn't come close to hitting).

I suggest the authors of this study should throw HMP a worm. The results wouldn't be so pretty.

 

Complete nonsense. HMPA is not designed to stop malware, tho some components can stop certain malware action.
The feature tested is designed to stop remote code-execution exploit which abuses (mainly memory corruption) application vulnerability.

I really don't get see some ppl want everything in a program especially complementary program like HMPA, and this sentiment is not limited to HMPA, I feel this often.

 

As already noted, it's not pure exploit test nor real-world test as it doesn't include behavior blocking. I wonder why they used such half-baked scheme. In earlier corporate test they did real-world test and separate results to pure-exploit and dynamic protection, which I think is much better. Now in this new scheme we can't tell how each product is good in exploit prevention nor whole dynamic prevention.

That said, still we can get some insight from this half-baked test.
All high scored AVs (Norton, Kaspersky, Avast) have strong NIPS. Yeah, NIPS is signature based detection so is not capable to block 0day, however it doesn't mean NIPS is useless. NIPS shorten time gap btwn new vuln discovery and official patch. Some vuln can't be prevented by anti-exploit but can be by NIPS (kernel exploit, logic-flaw).

As to simulator test, it makes sense as it would be only way to test unknown 0day exploit, but sure, their "provided" test won't be persuasive in such forum where there're many knowledgeable and critical ppl, even when the exploit is publicly available and criminal may use it or similar. I don't doubt Malwarebytes can make PoC in which only MBAE can block.

Honestly, tho it's great HMPA can block ITW exploit, what I want to see is not this. I'd like to see continuous challenge against HMPA by skilled researcher. EMET have been challenged and improved much. MBAE also have been challenged tho doesn't reach EMET's degree. IMO, it is MUST-have for HMPA to be really bullet-proof product. Also, I want to know if all exploits in this test were blocked by HMPA's memory protection or application lockdown but not process protection.

@Mayahana
Trend would have earned better score if they enabled firewall tuner (IIRC, it's default-off.) which is basically NIPS. Same goes to nearly all exploit tests including MRG's corporate exploit test (again, NIPS is not included in default OfficeScan, it's an addon) except ones performed by NSS Labs where they allowed each vendor configure their product.

@guest
I'm sure Webroot wouldn't get good score as they don't have ANY NIPS component even in premium suite. It just means their protection philosophy is different, they're more focusing on malware blocking, same as Emsi.

@FleischmannTV, @Nightwalker I would feel the same if it was full real-world test, but as Fabian said it's not and Emsi currently don't have strong NIPS nor behavior-based anti-exploit, so I don't feel need for conspiracy theory to explain the result.
Generally, most "suspicion" can be reasonably explained if one have enough knowledge about technically how each product works and testing methodology. The problem is ppl tend to think good products are good in all aspect (there's no such product), and often speak before they read methodology.

 

I don't go by tests...I invest my time in a product, and beta test. Very simple!

 

Actually, the ROP chain and shellcode that we provided for the artificial zero-day attack defeats our HitmanPro.Alert 3 as well, when you run the attack inside a virtual environment or on non-Intel hardware. We went all the way to beat anti-exploit solutions, including our own.

 

So, how do I test that claim in my own system that is running HMP.A... I don't have a VM installed, and I don't want to.