Matousec: Proactive Security Challenge 64 (bits)

Discussion in 'other anti-malware software' started by fax, Jun 22, 2013.

Thread Status:
Not open for further replies.
  1. fax

    fax Registered Member

    Joined:
    May 30, 2005
    Posts:
    3,899
    Location:
    localhost
    http://www.matousec.com/projects/proactive-security-challenge-64/

     
  2. jmonge

    jmonge Registered Member

    Joined:
    Mar 20, 2008
    Posts:
    13,744
    Location:
    Canada
    this is for a termination test?they did very poor:D
     
  3. SLE

    SLE Registered Member

    Joined:
    Jun 30, 2011
    Posts:
    361
    Wow. These products don't detect laboratory things which have no real world relevance. GOOD.:D
     
  4. Frank the Perv

    Frank the Perv Banned

    Joined:
    Dec 16, 2005
    Posts:
    882
    Location:
    Virginia, USA

    Wow, Webroot UnSecureEverywhere Complete 2013 got hammered.

    Product Score 22%. Protection Level = None. And Not recommended.

    ---

    But that is an odd test... testing different types of products together... and the results are not similar to other professional testing organizations.

    What's up with all that?

    .
     
    Last edited: Oct 13, 2013
  5. jmonge

    jmonge Registered Member

    Joined:
    Mar 20, 2008
    Posts:
    13,744
    Location:
    Canada
    trying out Ashampoo® Anti-Virus:thumb:
     
  6. fax

    fax Registered Member

    Joined:
    May 30, 2005
    Posts:
    3,899
    Location:
    localhost
    They just testing different different end points than other testing organisations. Their results may have nothing to do with the real ability of a software to protect the users from current threats. Pleanty of discussions in this same thread about limits of this test methodology. Just take their results with a bit of caution... something many users here can't :D
     
  7. vojta

    vojta Registered Member

    Joined:
    Feb 26, 2010
    Posts:
    830
  8. EASTER

    EASTER Registered Member

    Joined:
    Jul 28, 2007
    Posts:
    11,126
    Location:
    U.S.A. (South)
    It drives me crazy. HIPS is been for me the crown jewel of interactive flawless protection for years, but with Windows 8 x64 not a single freelance or startup is anywhere to be found for this platform.

    If only some developer would write a x64 driver for the now forgotten but formidable HIPS EQSYSECURE it would solve the whole spectrum of dependable windows coverage i been missing since XP.
     
  9. Solarlynx

    Solarlynx Registered Member

    Joined:
    Jun 25, 2011
    Posts:
    2,015
    Am I right that these tests simulate the case when you run malware on your PC?
     
  10. fax

    fax Registered Member

    Joined:
    May 30, 2005
    Posts:
    3,899
    Location:
    localhost
    They say so... but still just tests no real malware... for example, WSA get a "none" in terms of their score but in practice with real threads the picture is rather different (see other tests). So, just take those results with care as they can be completely misleading.
     
  11. Solarlynx

    Solarlynx Registered Member

    Joined:
    Jun 25, 2011
    Posts:
    2,015
    I mean if these tests simulate that you run malware on your PC then it's rather severe tests for rather rare scenarios (I guess you don't have desire to run malware on you PC) and most probably you will be OK even with the AV/FW which failed the tests of this type.
     
  12. fax

    fax Registered Member

    Joined:
    May 30, 2005
    Posts:
    3,899
    Location:
    localhost
    Well, they are actually testing also security suites... but yes... failing those tests does not mean the product cannot protect the system in real life cases. ;)
     
  13. ZeroVulnLabs

    ZeroVulnLabs Developer (aka "pbust")

    Joined:
    Mar 5, 2012
    Posts:
    1,189
    Location:
    USA
    100% agreed!

    Additionally there are SmartHIPS and not-so-SmartHIPS (nosier). This test methodology favors the not-so-SmartHIPS.

    This comment is not against any of the products that rank well in the test (some vendors of SmartHIPS might add detection of leak tests just to look good), it is against the test methodology itself.
     
  14. guest

    guest Guest

    I really don't understand the critics unless you haven't even read the introduction of test, it quite clear what is the purpose of the test, what are they testing and the methodology.

    http://www.matousec.com/projects/proactive-security-challenge-64/

    This is not a malware test, a phising test, an antiexploit test...

     
  15. EASTER

    EASTER Registered Member

    Joined:
    Jul 28, 2007
    Posts:
    11,126
    Location:
    U.S.A. (South)
    Well said! And thank goodness for pointing to this IMO a very important comparison which also needs to be considered when such tests are published among vendors.

    Regards Easter
     
  16. ZeroVulnLabs

    ZeroVulnLabs Developer (aka "pbust")

    Joined:
    Mar 5, 2012
    Posts:
    1,189
    Location:
    USA
    The post referenced above by vojta explains it perfectly well:
    https://www.wilderssecurity.com/showpost.php?p=2192678&postcount=36
     
  17. guest

    guest Guest

    I know the difference btw BB and HIPS, for me matousec is an HIPS test and it's totally valid. Anyway some of the software tested can be killed by matousec samples, or the OS can be broken... so it's a design fault a not a problem of a BB not detecting a synthetic test as dangerous, because actually it's "dangerous" no matter how "synthetic" it is.
     
  18. fax

    fax Registered Member

    Joined:
    May 30, 2005
    Posts:
    3,899
    Location:
    localhost
    Valid is a strong word :D

    Software is tested according to 11 levels. If you do not pass one level you are not tested against the others. So, if a software fails level 2 then it will get a "none" score even if it could have passed level 3 to 11. We will never know!

    Rather a "peculiar" approach, like if AV-comparatives would stop testing an AV as soon as they fail to detected a virus or a group of them... Oh, well...
    Conclusions: even on HIPS you cannot know if a product with low score has really bad HIPS performance in reality.
     
  19. guest

    guest Guest

    I wouldn't lose my time either testing something that does not block the most basic test... It's an approach, this test only tell you how good the HIPS component is, any product with a dedicated an decent HIPS component will go through all the tests.

    Ok, is not the best methodology but is the only HIPS test in the "market"
     
  20. fax

    fax Registered Member

    Joined:
    May 30, 2005
    Posts:
    3,899
    Location:
    localhost
    True :thumb:
     
  21. ZeroVulnLabs

    ZeroVulnLabs Developer (aka "pbust")

    Joined:
    Mar 5, 2012
    Posts:
    1,189
    Location:
    USA
    For the same reasons that a leak test is not a valid test of a SmartHIPS or BB the same applies for "app killing" or "OS breaking" tests. Those individual techniques are not found in-the-wild by themselves as they are isolated laboratory experiments which have nothing to do with reality. Those techniques might be integrated into a malware which performs that action, amongst with many other actions. But that specific action by itself does not necessarily constitute malware as found in-the-wild. If it would you can be sure many vendors would have already added detection for those individual techniques.
     
  22. kareldjag

    kareldjag Registered Member

    Joined:
    Nov 13, 2004
    Posts:
    622
    Location:
    PARIS AND ITS SUBURBS
    hi,

    Does this test help the end user to choose the more appropriated product? No.
    Does this test prove in an incontestable way which suite is the most effective? No.
    Does the methodology put all the products in the same and equal start line? No.
    Does this test reflect what might occur in the wild? No.
    Does this test demonstrates the full potential of each product? No.

    Most of all, how can Matousec make recommendations statements only based on a restricted methodology?
    Bitdefender for instance is one of the most effective suite, but as the methodology does not include pattern match detection, neither polymorphism, oligomorphism, metamorphism, junk code insertion, signature faking, database corruption, bad sector, stealth code, then it gets quite poor results.
    Even if they are Russia self-oriented, i'd rather the restrictive methodology (one criteria only like stealth code detection) and tests done by antimalware.ru/anti-malware-test.
    One day, Matousec decided to make money with software testing, taking advantage of already available tests and methodologies (firewallleaktester, Niko, Guizmo and myself).
    Unfortunately, system expert HIPS tend to disappear (SSM,Process/Neova Guard, EQSsecure and co) because this is a niche market only recommended for advanced/expert users as their protection is user dependent.
    The major market of average users is based on silent security, with a minimum of user's interaction and dependency: signature file detection, cloud architecture, system hardening, encryption technologies are much more implemented than behavioral based detection.
    I believe that it is possible to demonstrate in an exhaustive way and methodology the whole potential of one product.
    But as said Eugene Kaspersky, comparative testing in general is an equation without solution.
    Then yes, just another test, no more, no less.

    Rgds
     
  23. guest

    guest Guest

    It's a HIPS test, period
     
  24. Solarlynx

    Solarlynx Registered Member

    Joined:
    Jun 25, 2011
    Posts:
    2,015
    o_O

    :(

    it scares me to death

    so many malicious inventions


    yes, actually all successful products are not user dependant

    a user, unless he is a geek, is reluctant to use the security stuff which asks him what to do
     
  25. vojta

    vojta Registered Member

    Joined:
    Feb 26, 2010
    Posts:
    830
    If by 'dangerous' you mean that some of them they can destroy your system, yes they are. But more in the sense of poorly written, buggy software: You execute it knowing the risks.
     
Thread Status:
Not open for further replies.
  1. This site uses cookies to help personalise content, tailor your experience and to keep you logged in if you register.
    By continuing to use this site, you are consenting to our use of cookies.