MRG Effitas Online Banking Browser Security Certification Project Q4 - 2014

Discussion in 'other anti-malware software' started by malexous, Mar 4, 2015.

  1. malexous

    malexous Registered Member

    Joined:
    Jun 18, 2010
    Posts:
    828
    Location:
    Ireland
  2. J_L

    J_L Registered Member

    Joined:
    Nov 6, 2009
    Posts:
    8,516
    Would love to see HitmanPro.Alert tested.
     
  3. itman

    itman Registered Member

    Joined:
    Jun 22, 2010
    Posts:
    2,969
    Location:
    U.S.A.
    Never have been able to comprehend their scoring and ranking methods. Panda fails one botnet test, SpyEye, and they are scored third worst?

    Then their is the graph itself. All products except MSE scored 95% or better. That is pretty respectable in my book. However, the graph visual representation would lead you to believe a much larger deviation.

    Finally there is the test methodology itself. Using appinit to inject .dlls has been around since Win 3 days. This can easlly be prevented by changing your registry to only allow signed .dlls which is the default on WIN 8 and above.
     
  4. Mayahana

    Mayahana Banned

    Joined:
    Sep 13, 2014
    Posts:
    2,220
    Deviation skewing via graph manipulation is manipulation nonetheless, and I agree. It's quite easy to manipulate the 'perception' of the results of something by simply adjusting the graphics to create a false perception of reality, this company is known to do that, and it's quite annoying. Personally I think MRG are a joke.
     
  5. itman

    itman Registered Member

    Joined:
    Jun 22, 2010
    Posts:
    2,969
    Location:
    U.S.A.
    The second .dll inject test method used was:

    Simulator Test - Windows Application Compatibility
    Financial malware developers always find new ways to bypass current protection technologies. One of the new techniques is to use the Windows Application Compatibility feature (Shims). This technique is a two-step process. First, the malware modifies the Windows Compatibility Database in a way that Windows will inject the attacker supplied DLL into Internet Explorer, then hooks (redirects) the API calls, where the password can be found in a buffer passed to the function as a parameter.


    A read on Windows program application compatibility feature is here: http://www.howtogeek.com/howto/10436/using-program-compatibility-mode-in-windows-7/ .

    Appears MRG was referencing a vulnerability that had existing in Windows that was patched in January of this year: https://technet.microsoft.com/library/security/MS15-001.

    So what are we talking about here .... an exploit. It's a given most conventional AV/AMs suck against exploits although some are improving. So if you want to protect against exploits use EMET or MBAE; I prefer the former.
     
  6. Yuki2718

    Yuki2718 Registered Member

    Joined:
    Aug 15, 2014
    Posts:
    1,257
    I think graph manipulation is not a problem, all other testing organization do this more or less and how to interpret graph should be taught in primary education.
    Sad if anyone don't have this very basic ability and fooled by graph.:(
     
  7. itman

    itman Registered Member

    Joined:
    Jun 22, 2010
    Posts:
    2,969
    Location:
    U.S.A.
    I believe Mayahana has a point. The graph on the x axis starts at 80%. Of course, this is going to exaggerate the deviations. If the graph started at 0% which it should have, then the deviations would probably visually not be noticeable. Or, all products tested excluding MSE had a 95% or higher confidence level.

    Actually, what should have been used is a tabular representation that showed number tests failed along with detail on each test and it's relative weighting factor.
     
  8. vojta

    vojta Registered Member

    Joined:
    Feb 26, 2010
    Posts:
    830
  9. itman

    itman Registered Member

    Joined:
    Jun 22, 2010
    Posts:
    2,969
    Location:
    U.S.A.
    In AV-Test's case, they award a maximum of 18 points as explained here: Home-user products must achieve at least 10 of the 18 points available and at least 1 point in each category in order to earn an "AV-TEST CERTIFIED" seal of approval.

    Ref: http://www.av-test.org/en/antivirus...2014/avg-anti-virus-free-edition-2015-144980/

    In the actual protection ranking, a maximum of 6 points are awarded; 3 points for each test category I assume. Then AV-Test computes an industry average percentage for each test category. They don't elaborate on where that figure comes from but I assume it's the average of percentage scores for all products in the test. Finally, they compare the individual product category score percentage to prior calculated industry category average percentage and assign up to 3 points based on that. In other words, if a product category score percentage was approximately the industry category average percentage, then the product would be awarded 1.5 points.

    In other words, there is a baseline used but it is the industry average score. MSE receives a score of zero by virtue of being the lowest percentage scored.

    It is also important to note that this is a relative versus an absolute test. If the industry average was 50% for both categories, it would indicate overall most of the products tested were substandard. In this test the industry average was 94 and 99% for the two protection categories. MSE scored 55 and 78%. The next lowest scoring product I believe was Vipre at 77% and 99%. And then it jumps to 94 and 97% for K7, the next lowest scoring product. Bottom line - the standard deviation in the protection category for all products certified was very small.

    The VB100 test is the best visually in my opinion since their graph shows how each product ranked absolutely to all products tested: https://www.virusbtn.com/vb100/latest_comparative/index . BTW - their graph starts at 50%.

    Here's a complete VB100 report for Win 7: https://www.virusbtn.com/virusbulletin/archive/2014/12/vb201412-comparative
     
    Last edited: Mar 6, 2015
  10. vojta

    vojta Registered Member

    Joined:
    Feb 26, 2010
    Posts:
    830
    So one still needs 7 paragraphs to explain a single graphic. Not that great, right?
     
  11. itman

    itman Registered Member

    Joined:
    Jun 22, 2010
    Posts:
    2,969
    Location:
    U.S.A.
    Mr. Kapersky had a serious falling out with AV-Test procedures a while back: http://eugene.kaspersky.com/2013/05/09/av-test-certification-devalued/ .

    And so have most AV manufacturers with one or more AV lab at one time or the other. Bottom line - all these tests are subjective, done for profit, and really don't adhere to established scientific principles since they lack standardization and peer review. As far as the labs overseer, ATMSO, this article sheds some like on their creditability: http://www.infosecurity-magazine.com/news/amtso-has-credibility-gap-for-anti-virus-testing/ . As are many things in the IT security industry since its inception - a lot of "smoke and mirrors."
     
  12. Rasheed187

    Rasheed187 Registered Member

    Joined:
    Jul 10, 2004
    Posts:
    8,024
    Location:
    The Netherlands
    I don't know if MRG is reading these topics, but ones again I have to say that I'm getting tired of their way of testing. I want to know how all these techniques and malware are blocked. Is it by signature/heuristics or is it by behavioural monitoring? That kind of stuff is interesting to me.
     
  13. Zoltan_MRG

    Zoltan_MRG Registered Member

    Joined:
    Apr 9, 2015
    Posts:
    14
    Our technique is not an exploit which has been patched. If you carefully read the Windows advisory and our description you can notice that these two attacks have nothing in common - except the Windows Application Compatibility module. The Windows vulnerability is a privilege escalation vulnerability. In our attack, the attacker already had admin level privileges. The Windows vulnerability is about modifying the cache - in our attack, we don't do things like this. Our attack still works on latest updated Windows versions.
     
  14. Windows_Security

    Windows_Security Registered Member

    Joined:
    Mar 2, 2013
    Posts:
    3,061
    Location:
    Netherlands
    Well that is what I call a realistic real world scenario. :gack:
     
  15. ArchiveX

    ArchiveX Registered Member

    Joined:
    Apr 7, 2014
    Posts:
    1,015
    My thoughts exactly...:thumb:
     
Loading...