MRG Effitas – Real World Exploit Prevention – March 2015 (sponsored by Surfright)

According to A-V Comparatives, their Real World Comparative does include exploit testing: http://www.av-comparatives.org/wp-content/uploads/2014/10/avc_factsheet2014_09.pdf .

So this might be the closest to an objective test that can be currently had. Also their results are significantly different than that given for the recent MRG report.

The thing I find very suspicious in the MRG report is Eset's Smart Security results; they just don't correspond with any other testing of the product. Also it could be very well that the archaic system configuration used in many of the MRG tests are just not supported and correspondingly protected by many of the main stream vendors anymore.

Finally in the MRG "artificial" exploit test which one has to strongly suspect was custom engineered for HMPA, only EMET and MBAE additionally blocked anything. Plus the test was done using FireFox exploiting a plug-in. I would love to see how EMET would perform against a similar exploit using IE.

 

While we're at it: @Zoltan_MRG, there's a typo in the version number of Kaspersky Internet Security. You obviously mean 15.0.1.415 and not 5.0.1.415. (Pages 30, 33 and 36)

And btw, since the text says all the test were carried out between 9th March and 27th March, 2015: version 15.0.2.361 has already been released since 3rd February, 2015. Not sure why you were using an older version.

 

I don't understand all the hype about this test, I don't see any great surprises. I also think it's fair of SurfRight to show that an older version of HMPA was not perfect. MBAE also gets a high score, and according to pbust some of the misses were fixed.

EMET probably performs less good, perhaps it lacks a couple of blocking techniques. And I have not read the whole report yet, but I wonder how the AV's blocked the exploits, was it with HIPS or signature? I only care about pro active detection (behavior blocking).

 

Then you probably don't understand the nature of testing.

Testing is supposed to be impartial, objective, and fair to all those tested.


Thank you.

-Frank

 

Yes, I think we all agree with that, but how many of these "sponsored" tests are perfect and completely fair? Like I said, I didn't see anything shocking, it's what I expected. Y'all acting like some products where seriously disadvantaged or favored.

And perhaps you guys are right about some stuff, but I like to focus on the big picture. Both HMPA and MBAE are good choices when you want to run a dedicated exploit blocker, that's the end conclusion.

 

The AV's didn't block them with HIPS or behaviour blocking. MRG only monitored if a new process has been started or loaded and according to the rules they've chosen it's game-over after this. So every warning/alert which happend after malware execution was ignored. So the test only was checking for how good the products blocked the exploits. But...

Some products were included which never state they have any exploit-blocking features included. If the whole point is to test their exploit-blocking capability this simply makes no sense. Why compare them to software like MBAE, HMPA or EMET then?

• Either you let those antimalware-suites use all their protection modules (sandboxing, HIPS, behaviour blocking, etc. - everything that comes in after malware execution) and compare them of their value in protecting the user from the ultimate evil, the payload.
• Or you only include products which are able to detect exploits and block/prevent their execution.

Everything else is just comparing apples and oranges.

 

Lesson learned from this test? That next time Surfright should subcontract another laboratory with a more critical review on the testing methodology and better reputation. More fair, robust and transparent results and a typical win-win situation (product is good and testing organisation gets visibility). Instead now this look like a lose-lose situation with a product that is artificially pushed up and a testing organisation that managed the test superficially… Not bad, they miss two birds with a stone

 

MBAE definitely adds meaningful protection. Norton's exploit protection heavily rely on IPS and browser protection (Generic Exploit Blocking technology). Tho browser protection may be able to block some unknown 0day, basically they're only effective against exploit against known vuln. Norton rely on UxP and application behavior lockdown against unknown 0day exploit, but I don't expect too much for them. To be honest, I don't know how effective they are as I can't test them―Norton always block exploits before they chime in by IPS or browser protection, so I need to generate real 0day to test which is beyond my expertise. Anyway, browser protection and lockdown only works against popular application (former for IE, Fx, Chrome. latter is problem, we don't know what apps are protected by lockdown) so if you use unpopular application, add MBAE protection for them.

Also note, Norton is not much effective about document scan. When document includes exploit, Norton is not much effective and this is why I use OfficeMalScanner (I bit care about its no update since end of 2013 tho). Not only based on my hand on testing, but I found some academic research paper about document exploit detection, and it confirmed my finding of Norton is not capable about detecting document exploit. To be fair, in my testing Norton properly blocked subsequent drive-by by either URL blocking or download insight when link is alive.

Sorry for nitpicking but that's not exact. Correct sequence is:
URL filtration>IPS>signature and heuristic scan>download insight>SONAR
In this context reputation is equal to download insight. I don't know what DNA means but if it means cloud classification by machine learning, it should be included in heuristic scan (Suspicious.cloud) and DL insight.

 

I know, and while they may all stops 9mm bullet (well, it's simplified example, I know too much about those military things...) and broken by .50 magnum or 5.56mm NATO rifle, still if one of them can block .44 magnum or even .357 magnum (9mm) still it have value. I want to know which is the hardest to bypass.

Did you correct Office 20010 bug in past report which I posted in a past MRG test thread?

AVC test surely includes exploit, but it also includes many non-exploit executalbes so can't be viewed as exploit test. Also do not speak about correspondence, when methodology differ, it's natural that they don't correspond and most often such difference can be reasonably explained if you know technical deteals about the products and carefully read methodology (tho I haven't seen really detailed methodology in any tests).

Only EMET have EAF+, tho HMPA have IAF, dynamic heap spray, and HW-assisted CGI and both of MBAE and HMPA adds StackExec and behavior protection. As HMPA & MBAE can add behavior protection when new bypass or new type of exploit (e.g. logic flaw) come, EMET also can add EAF+ module specification and ASR which I find useful. Like Kees, I also added some custom modules. Note a researcher who recently bypassed EMET and MBAE said EMET is much tougher than MBAE.

Both, some AV have NIPS (sig) and behavior-based anti-exploit (e.g. Norton UxP, Kaspersky AEP, F-Secure Deepguard 6.0, ESET exploit blocker, G-Data, Panda) but I suppose they are not as effective as MBAE or HMPA.

[EDIT] Panda added in last paragraph.

 

I agree with Rasheed, I feel you guys expect too much for the test idealistically, as if this is a comparative tests. The 2 major purpose a vendoer apply sponsored test is, 1) see how the product perform 2) to find problem in the product. If results was good, they may publish it to promote sales. If bad, they can choose not to publish it legally. And they can use feedback to improve their products, and MRG giving feedback is not the first time, Zemana got feedback and used it to improve their product. I know, your arugument is they re-tested after feedback. But remember, Surfright could choose not to publish first test and only publish 2nd test. If they did, we can't see those thing, I think it's much worse. But they didn't, because they don't need to hide it. I noticed it when I looked it, but didn't mentioned it cuz it's not matter. 2 versions of HMPA is clearly in the graph.

What required in tests are 2 thing, 1) don't do injustice like data manipulation 2) be transparent. We can't know if they test justly, and as to transparency every testing organization are not enough, AVC is best within it, but this MRG test is IMO relatively transparent.

 

I agree. If they decide to use this results for marketing purposes, customers will see that there were two versions tested and that they improved their defenses after they got results from earlier version. IMO it would be unethical if they would hide this fact, but since they've published both results, I don't see much harm.

 

I am currently preparing a demo of a HMPA bypass as a reaction to the report, stay tuned...

 

Responsible and not full disclosure, I hope?

 

Not full disclosure, just a screen capture.
(It also impacts EMET and MBAE)

 

Yes, I was mainly talking about "application lockdown" which it lacks with standard configuration I suppose. I don't really like EMET, it looks a bit complex.

 

I have experienced EMET as being too slow when EAF+ is enabled. Without EAF+ EMET is not causing any major slowdowns. Although EAF+ is a key feature against info leaks.

 

What I basically meant, was that I wondered if the payloads were blocked (from running) via anti-exploit techniques (HIPS) or with signature. What happens after execution is not relevant since this is no behavior blocking or containment test. But if AV's were not given a chance to block the payload with signature then it would be unfair indeed. I'm not sure if this was the case.

 

I don't think it was done with the intention to make them look bad. At the end of the day, AV's claim to be able to stop malware, it shouldn't matter how it's delivered. You can clearly see that AV's like Norton and Kaspersky perform quite well, it's because they have a special anti-exploit module. ESET was a bit disappointing since they have also been offering anti-exploit for a couple of years now. Bitdefender and Avast performed nicely. The ones that scored under 80% should up their game.

 

Exactly my point, MRG have been quite transparent, so I don't understand all the fuzz. I don't see what's so wrong about the "artificial test", it's simply a showcase for a feature that other apps lack, big deal. It wasn't even included in the main exploit blocking test.

 

A tool like ProcessMonitor was used to see wether the malware process was actually started.

 

OK, so I suppose that if the malware process was loaded and active in memory, without an AV (or anti-exploit tool) alerting about it, it was considered a fail? Sounds logical to me. Because this also means that if HIPS/anti-exploit was bypassed, the AV was apparently not able to stop it with a signature as well.

 

They definitely were able to use signatures or blacklisting. According to the text the following levels were counted as success for the products:

So actually simply blocking the URL would be enough. Which imo invalidates the test if they were really trying to test whether a product does detect (and prevent) exploits or not.
They even say this themselves:

It's not about how it's delivered, that's not the point. Anti-malware suites claim they will be able to stop malware, but with all enabled modules. That's why most of them use different layers.

I, for one, am only complaining about chapter 4, the "product comparism". Claiming to test HMPA "alongside twelve competitor products" even though some are no competitors is not fair. Some products simply have no exploit blocking abilities, so why including them?.
In fact I am completely fine with all the other stuff. The product review of HMPA is great. Making all the effort to actually develop a simple Firefox plugin and then exploit it, adding all the source-code is really impressive and I applaud them for their transparency and - mostly - for their methodology (especially if you compare it to the test by PCSL).

Nope. It was considered a fail if a new process (the payload) spawned. That's it. Any alerts afterwards were discarded.

 

Hum ......... I have used EAF + to monitor non-browser app .dlls with no performance hit whatsoever. Also I use IE and have had no performance impact while browsing.

 

No tests are perfect, but i've always liked what MRG has done. As it's helped make HMPA even better, that's a good thing.