MRG Effitas – Real World Exploit Prevention – March 2015 (sponsored by Surfright)

I'll admit to not reading all of it, and merely 'skimming'. Exploits aren't my focus to be honest, but it seemed like a part of this test was based on simulated exploits? I don't put a whole lot of stock in simulated 'things' because those simulations usually lack realistic variables. Not sure in this case though, I just haven't had much faith in simulations because they rarely parallel real world activities or results.

 

It depends what you consider 'simulated'. Most of the exploit techniques will still be the same.

 

They must have had a good time testing it, as the MRG report has stand-up comedy quality humor included: look for instance at the title: "Real world exploit prevention" with simulated/synthetic tests .

 

Testing an anti-exploit solution is not about burning a 100k zero-day. It's about testing it in a realistic scenario and that's something that has been done in the MRG report.
It even doesn't even matter whether you're using a two-year-old version of IE8 running on Windows 7 for testing purposes.

 

Yep running outdated IE8 on Windows 7 surely increases the real word relevance.

Although true, to block an exploit one does not have to stop all intrusions, blocking some critical steps in the chain of events, will also do the job, see also Fabian Woser's comment post#16 and Peter's comment

 

The following endpoint security suites, in the following configuration, were defined for this test:
a. No additional protection, this snapshot was used to infect the OS and to verify the exploit replay
b. Avast Internet Security 2015.10.2.2214
c. Bitdefender Internet Security 2015 18.21.0.1497
d. EMET 5.2.5546.19547
e. Emsisoft Internet Security 9.0.0.5066
f. ESET Smart Security 8.0.304.0
g. F-Secure Internet Security 2.15 build 361
h. HitmanPro.Alert 3.0.34.167
i. HitmanPro.Alert 3.0.34.174. This build has been created by the feedback from MRG Effitas, so SurfRight developers were able to improve the efficacy of their products.
....
​

Surprised nobody else brought this up.. so I will.

So, uh, I wonder why HPA had two shots at this test with two versions of the product?

I'm very surprised that MRG admitted that two versions of HPA were tested. I think this is the kind of stuff that testers mostly keep quiet.

It could lead one to wonder..... why were two versions of HPA tested when no other product got to test an updated version of the product?

In most tests -- as far as I know -- there is a cutoff date for product versions. Otherwise tests would never end..

One could think that HitmanPro.Alert 3.0.34.167, did not clearly win the initial test -- that the developers were given a chance to 'try again' with the updated version. One could think that Norton actually won.... until the new version was rushed in to 'fix things.'

I'm definitely not a member of the tin foil hat brigade. But if any other tested capability had an update in the test period, but their product was not allowed to come in and retest... that would make the final result fixed.

All tests were carried out between 9th March and 27th March, 2015.​

I'm not going to go look. But if I am understanding the presented data correctly -- if there is any tested product had a version update in that time period, the final result is BS.

And even if there were not competing product updates in that time period, there would still be no logical explanation I am aware of that would explain one product getting to test two versions, and apparently an updated version based on tester feedback during the test.

This build has been created by the feedback from MRG Effitas, so SurfRight developers were able to improve the efficacy of their products.​

So MRG tested, gave feedback, HPA made adjustments, and then won.

Wow, what a surprise.

I suspect that Norton won the actual test, and HPA tied for second with Avast & Kaspersky.

Again, the biggest surprise for me is that this information is published, allowing us to figure this stuff out. Maybe there is the fear that an employee would later spill the beans...


That is all.

-Frank

 

Let me rephrase my statement: If you're testing an anti-exploit solution then it's not worth spending two months of full-time development to write a Chrome zero-day just for proving that an exploit mitigation tools works.
Although I recommend that you take a look at the basics of browser related exploit development. You will find out that most of the techniques are comparable when it comes down to the ROP part.

 

I said in post #55 although true, to block an exploit one does not have to stop all intrusions, blocking some critical steps in the chain of events, will also do the job, see also Fabian Woser's comment post#16

Also true: this illustrates the fun they must have had writing the report: real world exploit prevention test, while some of the protections have no existing real world exploits yet for Chrome.

 

I would also like to see the latest MBAE 1.06 in the test cause it was improved in the meanwhile....

 

Well,
I have to say that there has been controversy in the past with regard to an exploit test that has been commissioned by Malwarebytes in which HMPA seemed to be one of the losers --> https://static.malwarebytes.org/assets/datasheets/2015-03-31/RCEMitigations.pdf

 

and Norton right at the top, or near the top as usual...

 

I've just noticed something weird: The "artificial zero-day test" results of EMET (page 36).

EMET 5.2.5546.19547 (default settings):
Original attack: blocked
Provided attack: fail

EMET 5.2.5546.19547 (Firefox and plugin container protected):
Original attack: fail
Provided attack: fail

So, in their test EMET 5.2 with default settings blocked the exploit but after they have added protections for "firefox.exe" and "plugin-container.exe" EMET lets it slip through? Guess this indicates some sort of bug in EMET?

 

I didn't mention it because I thought that It was pretty obvious, but is good to bring it up just in case. It's not fair but is sponsored by HPA....

 

No need to suspect anything, both versions are listed in the product comparison and detailed results, nothing secret there.

Nice find, strange indeed.

 

Although I do not know if this is the result of very good proactive defenses, good script blocking or both.

 

Total package.. IPS, Sonar, Reputation, Insight, Firewall, working in concert.

 

Thanks @FleischmannTV, and you too @Peter2150, for your explanations.

Would you say there's any value in running HMP.A with a product like Emsisoft given the fact it has been known to step in first when enabled?

 

I'm sure FleischmannTV will respond to you, but EAM and EIS from Emsisoft are two completely different products than HMPA. It's like comparing apples to pumpkins lol HMPA is not an antivirus even though it has some detection capability it has nowhere close to that of an AV. HMPA is primarily for exploits, but can also protect against banking trojans, cryptomalware, loggers, and executions from USB devices. EAM is an AV that can detect a broad range of threats. HMPA is primarily focused on detecting, and blocking exploits. I would say it's next strongest attribute is blocking cryptomalware, and banking trojans.

 

Thank you for reporting the bug - it is a bug in the report, as the results were the other way round. We replaced the original PDF with the fixed one (because of hot-linking, the filename has not been changed). It might take some time still the caches purge, and also, delete local browser cache to see the changes.

 

@Zoltan_MRG

Let me round up the hilarious ethics and test procedure used by MRG

1. Using a real world test without Chrome exploits (page 43 of your report), and artificial exploits which not yet have been found in the real world for Chrome (post #58 ) or require a special executable to run to determine offset and compile the shellcode, referring to a technique which could be applied similar to a real world exploit patched in IE8 (page 35/36 of your report)

2. Testing a competing product when default settings don't include the program you test it with, so it surprisingly fails the test (post #70)

3. Testing an older version of a competing program (post #32), so it fails more tests

4. When sponsor submits a version, you test it, provide feedback so sponsor can adapt program to pass test (post #56)

5. Narrowing succes/fail definition so protection of other tested programs won't kick in and they fail the tests (post #16)

6. Using a special test which only works on certain hardware, designed and supplied by the sponsor (post #14)

Have a look at the business ethics of other safety testing organizations http://www.iecee.org/cbscheme/html/cbcode.htm.

 

1. Determining offsets can all be done dynamically, as long as you have RW access to memory on a 32 bit browser.
2. I agree with you on that part.
3. Running a few hundred tests takes some time and you can't switch versions during the testing (Yes, I am aware of the case where Surfright improved HMPA purely for better testresults)
4. Yes, that should no happen.
5. This test was only meant for exploit mitigation capabilities, waiting an hour before AV would notice that an application is mining bitcoins is beyond the scope of this test. (Although they also tested AV software which does not claim to have special exploit mitigations besides generic rules, MSE, Emsisoft, etc)
6. Eventually you can extract all ROP gadgets dynamically, although that's not an ideal situation when performing a pop-copy to the stack, because you have to extract quite a number of different ROP gadgets.

I also like to remind you that Malwarebytes also commissioned an exploit test in the past wich used a beta version (!) of HMPA (3.0.12.73) which was far from finished and in that test they also didn't test any Chrome exploits --> https://static.malwarebytes.org/assets/datasheets/2015-03-31/RCEMitigations.pdf

BTW, if you wan't to test HMPA, MBAE or EMET with a Chrome exploit, the exploit that VUPEN used during pwn2own 2014 to target Google Chrome can just be downloaded. (AV detection was 0 btw)

 

@ropchain

ad 1: Yes, as long as you have read/write access to memory, but that R/W access does not happen automatically. You first have to open another door. But that technicality is not the point I was trying to make: is the real world test irony of this synthetic test: it protects against a non-existing exploit, which success depends on leaking/determining the offset shown to be possible by an exploit patched long ago (as far as I know this specific mechanism has not been applied again in the wild).

ad 5: Your missing the point: Not the AV, but the HIPS element would have stopped some of exploits on the spot (not an hour later), see Fabian Woser post confirmed by Peter2150 (post #16 and 17 in this thread)

ad 6: Your missing the point: a test organisation uses a PoC provided by the sponsor, see the testing ethics link in previous post.

Re: Your reminder regarding MBAE test by PCLS, I agree on that, as I also made clear in that PCLS thread

Re: Your remarks in regard to MBAE vs HPMA. Just look at this, this and this post, those are not the posts of a surfright basher. So let's please keep discussion away from fanboy-ism. Would be better when Zoltan_MRG would respond on those six points (to prevent MBAE vs HPMA discussion, I directly addressed the post to him).

 

Couldn't have said it better myself.

On the other hand, people have to starting wising up on what these single vendor test's are all about. There are two types; a vendor who sincerely wants to verify the security worthiness of their product for internal use only and the one who want the use the tester as a proxy to promote his product. Obviously the financial incentive to the testor in the later case is to present the product in the most favorable light possible.

 

Well said. That makes sense, too, along with the timing of HMP.A coming out of beta and releasing it's first stable build. From a marketing perspective, that's brilliant timing. Although, when security-minded folks are easily able to poke many holes through the testing methodology, that does not look great. But quite frankly, the majority of the market will not see that nor understand it.

Personally, I really like what MBAE, HMP.A and EMET are doing, engineering toward a proactive goal against malware. And it's fantastic to have competition because that is important. I am not big fan of the one-sided sponsored tests in particular. I would absolutely love to see more anti-exploit testing but from a balanced and fair perspective.