AV-TEST Advanced Threat Prevention Test Results

Discussion in 'other anti-virus software' started by Minimalist, Feb 13, 2017.

  1. Minimalist

    Minimalist Registered Member

    Joined:
    Jan 6, 2014
    Posts:
    14,384
    Location:
    Slovenia
    https://pages.cylance.com/2017-02-08-CNT-AV-TEST-Report-2017-2092.html
     
  2. Minimalist

    Minimalist Registered Member

    Joined:
    Jan 6, 2014
    Posts:
    14,384
    Location:
    Slovenia
    http://www.csoonline.com/article/31...ylances-new-testing-methods-with-av-test.html
     
  3. boredog

    boredog Registered Member

    Joined:
    Feb 1, 2015
    Posts:
    2,499
    OH BOY Here we go again!!!!!!!!!!!!!!:eek:
     
  4. itman

    itman Registered Member

    Joined:
    Jun 22, 2010
    Posts:
    8,396
    Location:
    U.S.A.
    Can't get real excited about this one since it was a private commissioned($$$) test. If and when these "accommodations" make it into their public tests is when "it will hit the fan."

    AV-Test also is one not to shirk from controversy. Remember they were the outfit that developed "special testing" procedures for MSE back in 2013 when Microsoft raised a stink.

    -EDIT- I guess most notable in the CSO article is how Cylance after not getting the results they wanted from MRG and A-V Comparatives went AV Lab shopping for "friendly territory" and apparently found one.

    Also a bit of a puzzle to me is why didn't Cylance employ NSS Labs to do their testing? It is U.S. based like Cylance. It has by far the most advanced test platform to test 0-day malware. Finally, it specializes in testing commercial security products; notably commercial security network appliances employing AI technology and uses an established methodology to do so.
     
    Last edited: Feb 13, 2017
  5. entropism

    entropism Registered Member

    Joined:
    Dec 9, 2004
    Posts:
    475
    Would have loved to see Cylance put Webroot in the test, considering they're similar products with similar methodologies. Would have been more fitting competition.

    Edit: Reading the report here makes Cylance seem pretty unprofessional and petty, to say the least.
     
    Last edited: Feb 13, 2017
  6. Osaban

    Osaban Registered Member

    Joined:
    Apr 11, 2005
    Posts:
    5,352
    Location:
    Milan and Seoul
  7. Triple Helix

    Triple Helix Specialist

    Joined:
    Nov 20, 2004
    Posts:
    13,037
    Location:
    Ontario, Canada
    More and more the truth comes out. Anti-Malware testing is just a big money grab, never mind the methodology used, just throw it out the Window(s).
     
    Last edited: Feb 13, 2017
  8. illumination

    illumination Guest

    These testing organizations all use one testing methodology to test many products that are designed to handle threats differently, there is no way to produce accurate results that way, the tests need to be wrapped around the product and how it is designed to cope with threats not the other way around, not to mention it has been long known that most of them can be "greased" quite easily to obtain the desired results. It is why I test products for myself and draw my own conclusions.
     
  9. ProTruckDriver

    ProTruckDriver Registered Member

    Joined:
    Sep 18, 2008
    Posts:
    1,312
    Location:
    USN Retired 1969 ~ 1992
    If you go back and read all of my posts about testing, I never believed in it. The Truth Comes Out. :D
     
  10. SSherjj

    SSherjj Registered Member

    Joined:
    Mar 4, 2014
    Posts:
    174
    Location:
    New York, USA
    I am with you @ProTruckDriver ...one can't believe any of these tests. It's all false propaganda. in one way or another.
     
  11. Triple Helix

    Triple Helix Specialist

    Joined:
    Nov 20, 2004
    Posts:
    13,037
    Location:
    Ontario, Canada

    Methodology is out the Window. Who is in control?

     
  12. plat1098

    plat1098 Guest

    A snippy-snip from another thread. It's all about the CASH.
     
  13. Rico

    Rico Registered Member

    Joined:
    Aug 19, 2004
    Posts:
    2,080
    Location:
    Canada
    All these magazines, Ford's better, no Chevy. Truth is you can't say who is better, one person swears this one is best, while another swears at that choice. AV testing is a creative way to employ people & make $$$, which helps sales in the industry. Practice safe hex, & pretty much all Av's are equal. Perhaps AV testing is good at IDing the outliers (very bad or good), as always outliers are not the norm. The bigger question becomes free vs paid?
     
  14. illumination

    illumination Guest

    AV testing "when formed around the product and how it is designed" generally produces unforeseen bugs and vulnerabilities, it is quite useful to do so when not done with all businesses first priority, and that is making money.
     
  15. Osaban

    Osaban Registered Member

    Joined:
    Apr 11, 2005
    Posts:
    5,352
    Location:
    Milan and Seoul
    I don't think that because of Cylance and what has allegedly happened one should generalize and assert that all tests are biased or pay to play. Sure no test can be absolute with its results, but it certainly gives an indication in the long run about the overall performance of a product. Kaspersky and Bitdefender have had a remarkable performance over the years, Avira in the last 3 years has joined the top tier of high detection.

    We all know that Symantec (arguably still the biggest AV company) has withdrawn from AV Comparatives because they don't want their product to go through all the mandatory tests, even though they have always been excellent with the dynamic test. If anything this is real evidence that they cannot buy results and I don't think that they are trying to.

    Most AVs are quite reliable nowadays, but personally I wouldn't even trial an AV that doesn't participate in any collective test for any reason. Sponsored single tests are obviously ridiculous...
     
  16. guest

    guest Guest

    Cylance is the combination of a heavily marketed and "not better than average" technology funded by CIA and other "investors" so basically a free backdoor to every systems using it.
    Investors want results and investment-returns so Cylance must show itself better than everybody, which leads to that shady method.

    This report show it all, hiding their own weakness by disabling features of competitors with hidden methodologies. AV-Test surely grabbed a lot of cash on this contract... :rolleyes:

    I wonder how users of Cylance can still trust it and keep using it...unbelievable...but if they are happy with it so let them be...Anyway their system is not mine so i won't care much. :D
     
  17. zfactor

    zfactor Registered Member

    Joined:
    Mar 10, 2005
    Posts:
    6,103
    Location:
    on my zx10-r
    i do think some of the testing is bs personally but i also think some are pretty legit. i did get to meet the avc guys while in prague and they do for sure seem genuine and they fully explained their methodology while there to everyone. for me still the only testing i trust 100% is my own testing i do and real world use lol.

    imo this test is total bs.
     
    Last edited: Feb 14, 2017
  18. plat1098

    plat1098 Guest

    The security software industry has the giant surge in ransomware to thank for raising the bar greatly for better technology in detection and prevention. And like ransomware, the fundamental driving force is revenue. It's not mean and cynical, but a neutral observation. And many regular users aren't inclined to research too much, buying into any source that looks official and important. Like everything else, you have to sift thru a lot of puffery. Agreed: real life testing carries the most weight for a given user, how could it not?
     
  19. vlk

    vlk AV Expert

    Joined:
    Dec 26, 2002
    Posts:
    621
    I can't believe how incredibly flawed this test from AV-TEST is. I mean, I would have never believed that the guys over at AV-TEST would go this far and undersign something that is so obviously WRONG. They just did something terrible for their reputation if you ask me.
     
  20. itman

    itman Registered Member

    Joined:
    Jun 22, 2010
    Posts:
    8,396
    Location:
    U.S.A.
    What the AV Labs do is develop a "baseline" methodology to test the effectiveness of AV security products. Is it the most comprehensive testing that could be employed - no. To do so would be cost prohibitive. The labs also do not test for product reliability or vulnerabilities; one of my complaints. For example, how easy is it for malware to bypass the security product by disabling it. The only one who does something in this regard is Rubenking over at PC Magazine in his ad hoc testing.

    In specific to the methodologies the labs use, they are developed with uniformity as the top criteria. That is testing that is applied equally against all products i.e, a "standard." It is the only method for which all products can be evaluated impartially and fairly in comparative analysis. Once "exceptions" are created to factor in unique or special product features, all the preceding factors are lost. My opinion is if labs what to create "special" product tests that is fine. Just restrict those to the particular product being tested without any conclusions being drawn as to other security product effectiveness. In AV-Test's case, this would be a simple statement that Cylance was certified by them along with the applicable test results. And most important, methodology details that were employed for the testing.
     
    Last edited: Feb 14, 2017
  21. Cache

    Cache Registered Member

    Joined:
    May 20, 2016
    Posts:
    432
    Location:
    Mercia
    Agreed but you can put any old rubbish in there and still get the report!
     
  22. Cache

    Cache Registered Member

    Joined:
    May 20, 2016
    Posts:
    432
    Location:
    Mercia
    Not sure I would agree that they are similar. Cylance make a big thing about how they are superior when operating offline and some of the tests they applied specifically were offline tests.
    This extract from the executive summary:
    "In all test cases CylancePROTECT® showed extremely high efficacy prevention rates. They have a very
    reliable approach that works offline, without the need for regular updates even before execution of
    the malware. It also shows the dependency for the other products on regular updates, cloud queries
    or dynamic analysis".

    Webroot is far superior in my opinion and tests like these have no credibility.
     
  23. itman

    itman Registered Member

    Joined:
    Jun 22, 2010
    Posts:
    8,396
    Location:
    U.S.A.
    Which is a "disservice" to other AV products that employ behavior analysis and the like. These products first monitor for abnormal behavior. Once detected, they access the cloud to perform reputational analysis to minimize false positives and user interaction. On the other hand, many AI based products will just alert and leave the decision up to the user.

    -EDIT- Also noteworthy is reputation analysis is not exclusively cloud based. SmartScreen for example employs off-line lists that are updated frequently, an hour or less. Many AV vendors likewise have off-line lists they employ.
     
    Last edited: Feb 14, 2017
  24. mWave

    mWave Guest

    This review is unreliable, very flawed like @vlk has said... If people really want to use CylancePROTECT and believe they are better protected compared to using a normal security product then good for them, but I certainly wouldn't recommend relying on this product.
     
  25. itman

    itman Registered Member

    Joined:
    Jun 22, 2010
    Posts:
    8,396
    Location:
    U.S.A.
    One additional point that needs to be stated. Do not equate vendor endpoint products to their retail ones. They are not equivalent. Endpoint products are designed to be custom configured by internal IT security staff to their particular operating environment. Their "out-of-the-box" configuration per se is minimal security protection at best.
     
  1. This site uses cookies to help personalise content, tailor your experience and to keep you logged in if you register.
    By continuing to use this site, you are consenting to our use of cookies.