Malwarebytes Anti-Exploit

Why did Malwarebytes insist on includning a product which isn't even in the beta stage? What's the purpose except to bad mouth it? Please do explain.

No, the test was to compare MBAE to other products? Otherwise you wouldn't have other products in the first place:

When SurfRight contacted the research lab about their test result, why wasn't their results removed? I can only see one reason and that is to make them look bad, even though the product is still in alpha.

 

Again, the response is at the top of my previous post:

If we only wanted to bad-mouth HMPA3 we wouldn't have spend money to test all the other products. And as mentioned elsewhere, we didn't see the results until the test was finished. In fact we were expecting results to be much more positive for HMPA3 & EMET and were surprised with the actual results.

 

That's because you chose to include a product which isn't even in beta stage... bad business ethics I call it.

 

This discussion is going in circles. The response is above. You can choose to agree or disagree with it, but the response is there.

 

That's because you can't provide an answer.

Still no answer on why not include a potential 100% scorer (AppGuard) to the test.

 

Yes it's right there:

 

AppGuard definitely wouldn't have reached 100%, I can tell you that much, because it doesn't have any mitigations against reverse shell only payload types. It prevents guarded applications from manipulating the memory of other applications and has an anti-executable part as well. From what I understand this doesn't prevent remote code execution as well as some specific payload types.

 

Hi Pedro,
One remark and one question regarding your previous post.

This is not a bug, it hasn't been implemented yet because we are still developing it. Between CTP1 and CTP2 we added more applications (as mentioned in the release notes), and in CTP3 there will again be more software added to the software radar.

Could you provide us with the specific details and Metasploit parameters on how you bypassed our development build? We'll send you a beer if we can reproduce it using your details

Thanks!

 

I believe they took the following text from the CTP2 release notes as reference for their criteria, but it's best you contact them directly for clarification:

As you can see from the image it is not Metasploit. In the methodology it explains that some are private PoCs. These do not come from us (nor PCSL for that matter as I later learned) and we do not have permission to redistribute. Always happy to be bought a beer though

 

Ok, this is goot. This is the type of information many of us are interested in.

Even though it is speculation, there is a basis for the statement.

Can you or somebody with knowledge comment on how NVT EXE might have done with that testing methodology?

 

MBAE is in an increasingly competitive security niche. Ergo, they are competing. Good.

 

I'm personally a user of MBAE, but this test scandal lowers my trust in Malwarebytes.

 

Yes, I have been asking for such a test for a longtime, so I do appreciate it, it does give you an indication of how strong protection really is. On the other hand, it´s kinda weird that in another test, Kaspersky came out on top, not sure what to think of it.

https://www.mrg-effitas.com/mrg-eff...curity-exploit-prevention-test-february-2014/

 

Perhaps the MRG test didn't include payload types which don't try to establish a permanent infection on your computer, like reverse shell for example. I think Kaspersky is more focussed on those which try to deliver malware.

 

Yes I agree, apps like EXE Radar and AppGuard will probably not get high scores when "advanced" exploits/payloads are used. However, since AppGuard claims to be able to protect against "drive by attacks", it would be cool if they were included in the test.

 

If the exploit has a payload then AppGuard should block it. This is the MBAE thread though so I will limit my post about AG to this one post.

 

From my understanding not all payloads are executables files which are dropped on and executed from the hdd, hence anti-exe would fail here. PBust could elaborate on this because he has a much better understanding of these things.

 

Regarding other tests there's not much to go by. The only non-vendor sponsored tests seem to be the two from NSSLabs from Oct 2012:
https://www.nsslabs.com/reports/2012-consumer-avepp-comparative-analysis-exploit-protection
https://www.nsslabs.com/reports/2012-consumer-avepp-comparative-analysis-exploit-evasion-defenses

However these tests focus almost exclusively on download+exe type payloads and since most AVs have signature detection for stock Metasploit payloads they get high detection rates. Even the evasion test only looks at basic tricks such as compression and encoding but it's still download+exec.

I wish there were more anti-exploit tests that looked at more than just basic download+exec, but there's none that I know of, ergo the PCSL test.

 

Nice, I liked the first report.

But it wasn´t clear to me how the exploits were blocked, was it by signature/heuristics, or HIPS?

 

Yes that might be it.

@ ZeroVulnLabs

You forget to put ViRobot APT Shield to the test.

http://www.aptshield.net/

 

That's the problem with the little (and basic) testing being done with exploits, there's no way to tell the difference.

That's exactly the reason why we commissioned PCSL in order to answer that question (reactive vs proactive detections) by using different non-standard payload configurations.

 

I await HMPA to continue their improvements and perhaps turn the table on MBAE, this may turn into something Malwarebytes will regret someday, seems a very unprofessional approach from what I have seen...I will however continue to use MBAE free and wait until both parties have finished development and then decide who I'll buy from

 

@ZeroVulnLabs: Can you also answer this part of shadek's questions? Or in other words: What's you reasoning behind not taking care that HMPA3 gets removed from the test-report?

To give you an example why the current solution imo is not enough check out this news-site (PCMag.com) for example. This definitely is bad press for Surfright. Yes, somewhere inside that text there is a link to the "PC Security Labs"-website and from there one could download the updated test-report with the added red note. (At least as long as it stays on the frontpage otherwise it will be harder to find on that website.) But you can't see the note in the news-article. The article will stay online and already got indexed by Google a lot. Same for this really bad looking chart (for Surfright).
The damage is done. Bad reputation is the result. So if your following statement is really true:

Why don't you make sure Surfright gets no harm caused by this story and make sure PCSL IT really removes HMPA3 from the test?

 

Full reasoning here:
https://www.wilderssecurity.com/threads/malwarebytes-anti-exploit.354641/page-37#post-2399786

Specifically:

 

If you paid for these tests for the reasons you've stated "our objective was to see the differences in protection from proactive vs reactive approach to exploit blocking (i.e. signature-based vs proactive exploit mitigations) so we were surprised with the results as well."...Couldn't you have asked for the results to be given to you only and requested that they were not to be published to the public in general?