PCSL: Remote code execution exploit mitigations for popular applications (Aug 2014)

Discussion in 'other anti-malware software' started by FleischmannTV, Aug 12, 2014.

  1. FleischmannTV

    FleischmannTV Registered Member

    Joined:
    Apr 7, 2013
    Posts:
    1,071
    Location:
    Germany
    Test has been commissioned by Malwarebytes.

    ~ Copyrighted Image Removed - See PDF for Chart ~


    Source:

    -http://pcsl.r.worldssl.net/report/exploit/rce_mitigations_201408_en_malwarebytes.pdf
     
    Last edited by a moderator: Aug 12, 2014
  2. ZeroVulnLabs

    ZeroVulnLabs Developer (aka "pbust")

    Joined:
    Mar 5, 2012
    Posts:
    1,171
    Location:
    USA
    Since I'm guessing this will generate some discussion, please keep in mind the following disclaimer:
    In fact during the test we disagreed on a couple of points of the testing methodology and used payloads but we respected the views of PCSL which are what is reflected in the final report.
     
  3. siketa

    siketa Registered Member

    Joined:
    Oct 25, 2012
    Posts:
    2,697
    Location:
    Zagreb, Croatia
    Would like to see AppGuard tested.
     
  4. Impet

    Impet Registered Member

    Joined:
    May 5, 2013
    Posts:
    895
    Well done Malwarebytes. :thumb:
     
  5. SweX

    SweX Registered Member

    Joined:
    Apr 21, 2007
    Posts:
    6,429
    Thanks :thumb:
     
  6. Rasheed187

    Rasheed187 Registered Member

    Joined:
    Jul 10, 2004
    Posts:
    8,046
    Location:
    The Netherlands
    Wow shocking stuff! Some companies got some explaining to do. For example Kaspersky and ESET both offer a dedicated anti-exploit module, and they did quite bad. I would also like to know why HMPA3 did even worse than EMET. And of course kudos to Norton and MBAE. :thumb:
     
  7. Minimalist

    Minimalist Registered Member

    Joined:
    Jan 6, 2014
    Posts:
    5,078
    ESET has released v.8 BETA which should improve exploit blocker.
     
  8. markloman

    markloman Developer

    Joined:
    Jan 25, 2005
    Posts:
    433
    Location:
    Hengelo
    With disbelief we took notion of this comparative test. I will not go into details but we filed a complaint and insist on removal of our beta software from their report. Because even though this report states that they downloaded the software from the official website, our software is not even released or on our website. That is because our software is still in development, attesting the preview builds that we only post here on Wilders Security. Our preview builds only includes functionality that we would like to have tested. Our software clearly states Preview and the release notes state that it is not for production environments.
    In addition, the test was not done carefully. For example, PCSL did not add the standalone QuickTime Player to Alert's Exploit Mitigations while testing CVE-2012-0663, which the tester did not forgot while testing EMET.

    Since we did not ask PCSL to include our experimental software nor gave them permission to put it up against production software, we can only belief that their client insisted on including it for no other reason than slander.
     
  9. RubbeR DuckY

    RubbeR DuckY Developer

    Joined:
    Jul 7, 2006
    Posts:
    215
    Mark, slander certainly wasn't the intent.

    We wanted to put together methodologies that we thought made sense and have PCSL independently test them against as many anti-exploit products as they could. Out of curiosity, did you think any of the methodologies didn't make sense?

    I'll let Pedro chime in as well!
     
  10. ZeroVulnLabs

    ZeroVulnLabs Developer (aka "pbust")

    Joined:
    Mar 5, 2012
    Posts:
    1,171
    Location:
    USA
    The intent was to independently test MBAE and other products which claim to have some form of exploit blocking capabilities. All of the included software products do make some type of claim to protect against advanced threats and exploits. In the case of Hitmanpro.Alert3 there are in fact many more claims of exploit protection than from any other package included in the test (here & here).

    As for where the products were downloaded from, in the case of Hitmanpro.Alert3 the software was downloaded from your own server (http://test.hitmanpro.com/hmpalert3ctp2.zip) which is what I think they refer to by “website”. There are many pages on the web that point to this download link when searching for HMPA3.

    In regards to the test methodology, as mentioned in my post above, the methodology was decided upon by PCSL. Our only request to them was to test all software in its default configuration as well as to test newer and relevant CVEs with different payload configurations to test the ability of proactive exploit blocking without relying on signature detection. For example in the QuickTime Player vulnerabilities it seemed that the HMPA3 “software radar” didn’t pick up the installed QuickTime Player correctly and this might be the reason for the misses.

    There is another comment in the detailed results for HMPA3 we received from PCSL which says (google translate) the following: “CVE-2012-0507 session acquisition is successful, some functions can be used, considered failed”. In some cases for other products the reverse shell was created but then the connection was killed as per comments by the testers: “After a successful connection reverse_tcp load in a short period of time may be operable, but will eventually be cut the firewall, in this case considered to pass”. I personally would have been even stricter and considered the establishment of a reverse shell an automatic fail as the exploit was able to execute code on the target machine. But again, the testing methodology was decided upon by PCSL and not by us.

    Taking into consideration all of the above if you still want PCSL to take out HMP.Alert3 off from the report please confirm and we will talk to them about it.
     
  11. J_L

    J_L Registered Member

    Joined:
    Nov 6, 2009
    Posts:
    8,516
    There better be more updates and separate tests, still an incomplete picture.
     
  12. AutoCascade

    AutoCascade Registered Member

    Joined:
    Feb 16, 2014
    Posts:
    626
    Location:
    United States
    Why didn't you test their released software version 2?

    I mean you tested EMET 4.1 but surely 5.0 was available at that point since it's been on Microsoft's 'website' for some time now.

    I would like to have seen some Sandbox software tested also against the same exploits.
     
  13. subhrobhandari

    subhrobhandari Registered Member

    Joined:
    Nov 6, 2009
    Posts:
    708
    1. All the tests are executed on Windows XP SP3 Operating System in English, without any other additional patches. EMET 4.1 was used as EMET 5.0 does not support Windows XP. For me it would have been much better if the test was done in Win 7/8.

    2. In the Product Information page, Its only mentioned as "HitmanPro.Alert3". First of all, you should clearly mention this as a CTP. On a first glance someone who doesnt know ought to think this is the final product. There is no mention in the whole pdf that this is a CTP. The CTP version's link is not available from any of the browsing pages of the surfright website. It was posted on Wilders. Second, including a a CTP and a beta in a test while all the other players (MBAE beta is available on the official forum - open betas, but still you should mention as a beta) are stable versions is not a good gesture. Either include stable versions for all products or clearly mention in the information page this is a CTP version and things can go much differently in the final product.

    Mark said "our software is not even released or on our website.". RELEASED. No pages from surfright have the the download link listed.
     
    Last edited: Aug 12, 2014
  14. Frank the Perv

    Frank the Perv Banned

    Joined:
    Dec 16, 2005
    Posts:
    882
    Location:
    Virginia, USA
    Oh my.

    Where a product is released (and it is released -- as anybody can download a copy) and discussed at length on a public message board... it is unreasonable to object when somebody wants to test your claims or the efficacy of your product.

    Sure, there are testing issues and sometimes unfairness in the way testing is conducted.

    MBAM got raped in its depiction in this recent test:
    http://www.dennistechnologylabs.com/reports/s/a-m/2014/DTL_2014_Q2_Home.1.1.pdf

    It was simply ridiculous to graphically portray MBAM as the worst product in the test when MBAM was completely different from the other products (an A to B test).

    Claims of slander make you appear illegitimate. You need to simply make your objections known and drive on. And seek to have your product tested in future tests.

    It's 'funny' how product proprietors who complain loudly about a test often end up sucking in all the independent tests. Don't be that guy.

    If your product is good, this is just a minor bump in the road.


    That is all.


    -ftp
     
  15. Nebulus

    Nebulus Registered Member

    Joined:
    Jan 20, 2007
    Posts:
    1,582
    Location:
    European Union
    I checked the results for EMET, and I realized that all exploits that managed to get past it are JRE exploits. I didn't see anywhere in the methodology exactly what configuration was used for EMET (and to be more precise, if JRE executables were included in EMET's list of protected applications at the time of testing)...

    Later edit: checking the details of some of the exploits used, I realised that it is normal that EMET would not protect against this kind of threats...
     
    Last edited: Aug 13, 2014
  16. Windows_Security

    Windows_Security Registered Member

    Joined:
    Mar 2, 2013
    Posts:
    3,082
    Location:
    Netherlands
    Mark, let me first make clear that you are totally right that it is unfair to test beta software which is not released to a broader audience.

    On the other hand, as much as I appreciate your software, Surfright has applied marketing tactics which microsoft applied in their worst days: vapourware (see http://nl.wikipedia.org/wiki/Vapourware) before public release HMPA 3 has released a factsheet stating its superiority over competing products http://dl.surfright.nl/Alert-3/HitmanPro-Alert-3-Datasheet.pdf. When PDF was first published, MBAE was still in BETA!

    On top of that the HMPA released a test program to show that HMPA protected against actions the competing offers did fail to protect against: there is a saying that goes way back to comment on this: quod tibi vis, mihi fac, quod non tibi vis mihi non fac; non facias aliis hoc quod fieri tibi non vis
     
    Last edited: Aug 13, 2014
  17. markloman

    markloman Developer

    Joined:
    Jan 25, 2005
    Posts:
    433
    Location:
    Hengelo
    I think it is disrespectful and dishonest that you, the organizer, took one of our development race cars from the factory assembly pit, with its engine still in development, just for the satisfaction to see it fail against seasoned cars halfway through the racing season.
    We have respect for Malwarebytes, being a little contender like us, but pulling a stunt like this is very disappointing. I know things in the US go a little different but you guys are better than this. Your products are strong, you don't need this. So obviously, yes, we insist that PCSL removes our development preview of HitmanPro.Alert 3 from their report, that the graphics are updated accordingly and available for the press so they can update their articles.
     
  18. erikloman

    erikloman Developer

    Joined:
    Jun 4, 2009
    Posts:
    3,032
    Location:
    Hengelo, The Netherlands
    What intent did you have with putting up development software against production software?
    We are a really small company that do not have the man power to test all sorts of software security combinations. That is why we publish preview builds here at Wilders where the software clearly states it is a preview (see screenshots below).

    That press message was an announcement for RSA in California last February and only contains a few screenshots and a movie of the upcoming product in action – a very old build I might add. We wanted to attract other companies to come and take a look at a demo of the product that we have in development. That PDF is a leaflet for RSA.

    That hmpalert3ctp2.zip (yes it says CTP) is on a file server and none of our pages on our official website (www.surfright.nl or www.hitmanpro.com) link to that CTP file. Preview builds have always been published here on Wilders Security Forum only due to the many security enthusiasts here who are willing to try unreleased security software. You know how the web works, you post a link and some members copy it.

    The software radar does not cover all the applications yet. This is because Alert 3 is still in development. For example, Windows Media player was not yet recognized in CTP1, but is in CTP2. You could have taken the effort to add it manually though.

    Point is, our software clearly states that it is PREVIEW and PRE-RELEASE software as can be seen in the following screenshots:

    Alert3-CTP2-Preview-Text.PNG

    Alert3-ReleaseNotes.png
    From: https://www.wilderssecurity.com/thre...discussion-thread.324841/page-78#post-2393982

    See Mark's post above.
     
  19. markloman

    markloman Developer

    Joined:
    Jan 25, 2005
    Posts:
    433
    Location:
    Hengelo
    The datasheet was at that time (half a year ago) based on the current MBAE BETA version and the HMPA3 ALPHA version. As you can see in our early datasheet our Control-Flow Integrity (CFI) technology (which enables hardware features in modern Intel processors to assist in the detection of ROP attacks) is not listed either. The FEATURE list in the datasheet was created to attract companies to visit our booth at RSA.

    Our Exploit Test Tool was obviously built for testers to make compatibility testing easier, to verify if the preview builds of HMPA3 work correctly or if there is e.g. a loss in detection capability when users have other security software is installed. HMPA3 also takes care of other security software to make sure that their detection capabilities are not lost either. In effect you can also use our Exploit Test Tool to test if other security software is capable of blocking specific exploit techniques, just like Malwarebytes' mbae-test.exe exploit test tool.
     
  20. Krusty

    Krusty Registered Member

    Joined:
    Feb 3, 2012
    Posts:
    2,880
    Location:
    Australia
    ... And I'd like to thank Erik and Mark for giving us the opportunity to test their CTP programs.

    :thumb: *puppy*
     
  21. siketa

    siketa Registered Member

    Joined:
    Oct 25, 2012
    Posts:
    2,697
    Location:
    Zagreb, Croatia
    I like both products/companies but think this time Hitman guys are right.
    You can not compare stable and beta releases.
     
  22. Pliskin

    Pliskin Registered Member

    Joined:
    Feb 8, 2009
    Posts:
    341
    What if they tested HitmanPro.Alert 2.6.5, would that be OK?
     
  23. FleischmannTV

    FleischmannTV Registered Member

    Joined:
    Apr 7, 2013
    Posts:
    1,071
    Location:
    Germany
    HitmanPro.Alert 2.6.5 doesn't have any exploit mitigations.
     
  24. markloman

    markloman Developer

    Joined:
    Jan 25, 2005
    Posts:
    433
    Location:
    Hengelo
    According to the Exploit Mitigations Test report, HitmanPro.Alert 3 CTP2 did not stop many exploits. We tested some of the exploits that we apparently 'failed' and made a video. In this video you will see CVE-2012-0663 QuickTime, CVE-2012-4792 IE8, CVE-2013-3163 IE8 and 2 x CVE-2013-1488 Java7.

    The report does not give details on the used configuration, if it was a virtual environment or what kind of payloads were setup in Metasploit. So in the video you will see that the Metasploit exploit either tries to start the Windows Calculator or, in case of Java, initiate a Meterpreter Shell. The first 3 exploit attempts are blocked on the exploit technique and both Java tests are blocked on 'sandbox escape'.

    Enjoy the show: https://www.youtube.com/watch?v=4re2p-Yf8dQ&feature=youtu.be
     
    Last edited: Aug 13, 2014
  25. Rasheed187

    Rasheed187 Registered Member

    Joined:
    Jul 10, 2004
    Posts:
    8,046
    Location:
    The Netherlands
    I think that this was also a reaction to the comparison that was made by SurfRight earlier this year, between MBAE and HMPA3. It made MBAE look bad, so Malwarebytes returned the favor. But I would like to know why HMPA did so badly. Was the test not done correctly or fairly perhaps? How come that at least 8 more exploits are blocked when Mark Loman did the tests? :)
     
    Last edited: Aug 13, 2014
Loading...