PCSL: Remote code execution exploit mitigations for popular applications (Aug 2014)

That pretty well settles it.

You Hitman dudes did the first comparison of products.... with MBAE in BETA.

Then you complain when 'the other guy' then does a comparison?

My verdict is GUILTY.

You Hitman dudes can dish it out, but you can't take it.

So lets just stop with the stupid slander/libel talk... and get back to making great products.

We all look forward to AV-C or somebody good comparing the products in the future -- when both products are ready to roll.

Just next time include NVT EXE and AppGuard.


That is all.

My declaration is final.

Submit any appeals to Upside Down or Rightside Up or whatever his handle is.


Thank you,


-ftp

 

First there was 'Star Wars', and now the sequel 'Anti-Exploit' wars. Much ado about nothing.

 

Mark/Erik, our intent was obviously never to seek to criticize any one specific company, let alone one in a similar position as us. I have always respected the work done at Surfright and as you probably recall when you announced HMPA3 I mentioned it was great to see other vendors going into the space of offering proactive (rather than reactive) technologies against exploits as it helps validate the path we started on a couple of years ago.

I can 100% assure you that we are not trying to gain any satisfaction from “seeing you fail” – we honestly were just curious to see how MBAE stacked up against other products which are well known in the exploit mitigations space. If anything we expected clearer results showing the benefits of proactive exploit mitigations rather than reactive (i.e. signature-based AVs), so we were surprised to see the results as well.

With this in mind we have coordinated with PCSL to add a very clear and big note, right up front, in the official test document stating your point about HMPA3 being beta and unfinished. Updated link available at
http://pcsl.r.worldssl.net/report/exploit/rce_mitigations_201408_en_malwarebytes.pdf

As for the fails, I suggest you contact PCSL as they are the ones who decided the methodology, the ones who tested and the ones who decided the scoring criteria. We did not even know how HMPA3 fared until the test was finished. PCSL has done a good job and found issues with MBAE as well, which we appreciate from a QA perspective to improve the product.

 

I like your post pbust, hopefully this doesn't get into a flaming war as it does not need to.

 

Thanks, and agreed!

 

So a feature comparison sheet is in your opinion the same as a comparison test from a test organisation?

 

Malwarebytes did a real blunder on this one. They should apologize.

 

Erik,

Of course you are correct that they are not the same. But, they are not as dissimilar as you may be feeling right now.

They are both analytical processes that result in a product that shows capabilities of your own product vs. capabilities (or lack thereof) of other products.

-------

All that aside... you guys are handling this all wrong. This is why Bill Gates said that computer nerds always need good PR/sales guys to handle marketing. You have created a little Wilders mini-drama where you look overly defensive. None of this was necessary. Great products sometimes don't test well.

State your objections, let it go, and drive on. Worry more about the product of tomorrow than the test of yesterday.

You have done one thing well. You have created a lot of interest in a test in the future.

We look forward to it.



-ftp

 

@ZeroVulnLabs

The report doesn't state what kind of payloads are behind "a, b, c". Do you have any info on that?

 

They varied depending on the vulnerability, but were mostly execs, different forms of reverse shells, messageboxes, etc. The idea behind using different payloads was to see how "proactive" a block really was as opposed to if the block was driven by some sort of reactive signature of the stock Metasploit framework.

 

I wish AppGuard would have been included in this test.

 

Exactly, AppGuard should have been included in the test and in every other test that will come in the future.

 

Had AppGuard and NVT EXE been included, the soap opera drama would have only been better.

This needs to take on more of a professional wrestling flavor.

We need some good posturing and hyperbole.

And then we need to set up a Cage Match.

So the participants need to start thinking about what theme music they want played as they enter the ring.

Had the others been included, it could have been a tag team match.

 

Oh yes it does, thankfully!

There was already a post about this somewhere that I can't find now (this thread or another)...

AppGuard doesn't try to stop any execution/exploits, right?

I never tried it since what I found just sounds like an awful system-wide inconvenience. No, I don't want something screwing with programs and breaking stuff, configuring little things. I want to be able to install/update, run everything as admin and be able to do whatever without thinking about it. Exposed/online stuff that doesn't need that goes in Sandboxie, and likewise never thought about.

Finally, of course anti-executables are dumb and useless (if want to use a free one, in every sense, just SRP). Any and all code can execute in a process without ever being "executed." And then AppGuard will let exploits do most anything they want (payloads)? And it prevents other programs' memory from being accessed? So what, who cares? Causes more problems than it would ever help.

It's the most potentially-interesting program when I heard about it, but just awful to think about using... This coming from someone that LOVES a lot to configure/tweak, but only when it's worth it and doesn't create inconvenience. (Like NoScript: yuck, yuck, yuck. Not in a million years.)

 

Yeah, this has been a demonstration of how not to handle a dispute.

Here is an example of a dispute that was handled discreetly and without drama.
https://www.wilderssecurity.com/thre...nis-technology-labs-dtl-2014-q2-tests.366509/

I'm just guessing, but I think it was MBAM that quietly asked that one to be taken down -- as they got screwed in the graphical depiction product.

==========

But now that it has become a Wilders mini-drama, lets get back to talking about a computer nerd cage match to the death.

~ Removed Off Topic Link ~

----------

But seriously... both vendors are respected around here. Both are working in an area that needs good products.

Like bad gas, this too shall pass.

 

AppGuard should block any exploit that has an executable payload (I acknowledge that nothing is 100% full proof) . AppGuard should perform top of the line when it comes to blocking drive by downloads that enter the system by way of exploit. AppGuard was specifically designed to mitigate these type of threats. I think you are misunderstand the way AG works.

 

No, he isn't misunderstanding anything. Several payload types used in this test were of the category that AppGuard doesn't do anything against. I think you are assuming that payload equals drive-by download. This is not true. AppGuard neither mitigates remote code execution nor certain payload types.

 

No, i'm not misunderstanding. His post makes it sound as though AG does not mitigate exploits at all. You need to give credit where credit is due. I'm well aware that there are some exploit types that AG does not mitigate.

 

Btw.. I would say AG does mitigate the vast majority of exploits in the wild. Solely memory based exploits are not very prevalent.

 

Except that it was 'removed for review'. How long does it take to review and decide what to do with it? I've been waiting for the results of the 'review'. If the thread isn't going to be re-opened, it should say 'closed' with a note to say the matter is being dealt with privately, or removed entirely.

 

I don’t suppose there is a Professional Standard of Ethics in the anti-malware business…

I do see both sides of the issue... which have been extensively covered.

MBAM is now a well established company rapidly growing in products and assets. Hitman is just getting rolling. So I understand where Hitman feels unduly whacked by a big guy… and concerned about the potential impact of a negative test result on a beta product. MBAM can weather bad test results… as they have done well in many tests and have size and reputation already established. Hitman is still trying to break out. So an unfair test on a little company is pretty harsh.

What would be good is for both sides to agree to a couple of simple caveats, then do the proverbial electronic handshake, and move on.

Optimally, these companies could walk away from this respecting each other, or at least respecting the other’s willingness to attempt to play fair and exert effort to do the right thing.

A couple of really simple standards--

1. Agree to not use comparisons based on alpha/beta products on the website or in marketing materials.

2. Agree to accept the results of the testing of finished products without drama. Certainly, a company has the right to object to test methodology. And a statement should be made to indicate any shortcomings or misapplication of the test model. But a statement should be issued, and then everybody needs to drive on.

What else? What are other simple, reasonable, professional standards that could be asked?

What would be really interesting is a co-sponsored test of final products by a mutually agreed upon testing organization (like AV-C) and mutually agreed upon methodology. The costs of the test would be paid percentage-wise based on market value or gross revenue of said companies. But that is probably a bridge too far at this point.

Can we get reps from both companies to do a gentleman’s agreement to just a couple of simple guidelines? Not a whole long list... just a couple and just consummated by an electronic handshake.

It would be cool if we can get a proverbial electronic handshake from both fine companies out of this Wilders mini-drama. I will be renamed “Frank the Peacemaker” and everybody will live happily ever after.

 

Frank, neither one of the concerned parties have posted in this thread for over 24 hours, so maybe they have let it go, if people stop posting in this thread maybe things will quite down and slowly fade away... or maybe not.... who am I to know!!!

 

Hey, don't interfere with my important international peace initiative.

Let's see if for the good of the Wilders community, the good of the anti-malware community.... and the good of all humanity... either vendor or both decides to take the peace challenge.

If this works, I'll move on to the Israeli-Palestinian peace challenge.