AV-TEST Advanced Threat Prevention Test Results

https://pages.cylance.com/2017-02-08-CNT-AV-TEST-Report-2017-2092.html

 

http://www.csoonline.com/article/31...ylances-new-testing-methods-with-av-test.html

 

OH BOY Here we go again!!!!!!!!!!!!!!

 

Can't get real excited about this one since it was a private commissioned($$$) test. If and when these "accommodations" make it into their public tests is when "it will hit the fan."

AV-Test also is one not to shirk from controversy. Remember they were the outfit that developed "special testing" procedures for MSE back in 2013 when Microsoft raised a stink.

-EDIT- I guess most notable in the CSO article is how Cylance after not getting the results they wanted from MRG and A-V Comparatives went AV Lab shopping for "friendly territory" and apparently found one.

Also a bit of a puzzle to me is why didn't Cylance employ NSS Labs to do their testing? It is U.S. based like Cylance. It has by far the most advanced test platform to test 0-day malware. Finally, it specializes in testing commercial security products; notably commercial security network appliances employing AI technology and uses an established methodology to do so.

 

Would have loved to see Cylance put Webroot in the test, considering they're similar products with similar methodologies. Would have been more fitting competition.

Edit: Reading the report here makes Cylance seem pretty unprofessional and petty, to say the least.

 

Do I have to submit my private info to download a report? Not a good way to promote their product.

 

More and more the truth comes out. Anti-Malware testing is just a big money grab, never mind the methodology used, just throw it out the Window(s).

 

These testing organizations all use one testing methodology to test many products that are designed to handle threats differently, there is no way to produce accurate results that way, the tests need to be wrapped around the product and how it is designed to cope with threats not the other way around, not to mention it has been long known that most of them can be "greased" quite easily to obtain the desired results. It is why I test products for myself and draw my own conclusions.

 

If you go back and read all of my posts about testing, I never believed in it. The Truth Comes Out.

 

I am with you @ProTruckDriver ...one can't believe any of these tests. It's all false propaganda. in one way or another.

 

Methodology is out the Window. Who is in control?

 

A snippy-snip from another thread. It's all about the CASH.

 

All these magazines, Ford's better, no Chevy. Truth is you can't say who is better, one person swears this one is best, while another swears at that choice. AV testing is a creative way to employ people & make $$$, which helps sales in the industry. Practice safe hex, & pretty much all Av's are equal. Perhaps AV testing is good at IDing the outliers (very bad or good), as always outliers are not the norm. The bigger question becomes free vs paid?

 

AV testing "when formed around the product and how it is designed" generally produces unforeseen bugs and vulnerabilities, it is quite useful to do so when not done with all businesses first priority, and that is making money.

 

I don't think that because of Cylance and what has allegedly happened one should generalize and assert that all tests are biased or pay to play. Sure no test can be absolute with its results, but it certainly gives an indication in the long run about the overall performance of a product. Kaspersky and Bitdefender have had a remarkable performance over the years, Avira in the last 3 years has joined the top tier of high detection.

We all know that Symantec (arguably still the biggest AV company) has withdrawn from AV Comparatives because they don't want their product to go through all the mandatory tests, even though they have always been excellent with the dynamic test. If anything this is real evidence that they cannot buy results and I don't think that they are trying to.

Most AVs are quite reliable nowadays, but personally I wouldn't even trial an AV that doesn't participate in any collective test for any reason. Sponsored single tests are obviously ridiculous...

 

Cylance is the combination of a heavily marketed and "not better than average" technology funded by CIA and other "investors" so basically a free backdoor to every systems using it.
Investors want results and investment-returns so Cylance must show itself better than everybody, which leads to that shady method.

This report show it all, hiding their own weakness by disabling features of competitors with hidden methodologies. AV-Test surely grabbed a lot of cash on this contract...

I wonder how users of Cylance can still trust it and keep using it...unbelievable...but if they are happy with it so let them be...Anyway their system is not mine so i won't care much.

 

i do think some of the testing is bs personally but i also think some are pretty legit. i did get to meet the avc guys while in prague and they do for sure seem genuine and they fully explained their methodology while there to everyone. for me still the only testing i trust 100% is my own testing i do and real world use lol.

imo this test is total bs.

 

The security software industry has the giant surge in ransomware to thank for raising the bar greatly for better technology in detection and prevention. And like ransomware, the fundamental driving force is revenue. It's not mean and cynical, but a neutral observation. And many regular users aren't inclined to research too much, buying into any source that looks official and important. Like everything else, you have to sift thru a lot of puffery. Agreed: real life testing carries the most weight for a given user, how could it not?

 

I can't believe how incredibly flawed this test from AV-TEST is. I mean, I would have never believed that the guys over at AV-TEST would go this far and undersign something that is so obviously WRONG. They just did something terrible for their reputation if you ask me.

 

What the AV Labs do is develop a "baseline" methodology to test the effectiveness of AV security products. Is it the most comprehensive testing that could be employed - no. To do so would be cost prohibitive. The labs also do not test for product reliability or vulnerabilities; one of my complaints. For example, how easy is it for malware to bypass the security product by disabling it. The only one who does something in this regard is Rubenking over at PC Magazine in his ad hoc testing.

In specific to the methodologies the labs use, they are developed with uniformity as the top criteria. That is testing that is applied equally against all products i.e, a "standard." It is the only method for which all products can be evaluated impartially and fairly in comparative analysis. Once "exceptions" are created to factor in unique or special product features, all the preceding factors are lost. My opinion is if labs what to create "special" product tests that is fine. Just restrict those to the particular product being tested without any conclusions being drawn as to other security product effectiveness. In AV-Test's case, this would be a simple statement that Cylance was certified by them along with the applicable test results. And most important, methodology details that were employed for the testing.

 

Agreed but you can put any old rubbish in there and still get the report!

 

Not sure I would agree that they are similar. Cylance make a big thing about how they are superior when operating offline and some of the tests they applied specifically were offline tests.
This extract from the executive summary:
"In all test cases CylancePROTECT® showed extremely high efficacy prevention rates. They have a very
reliable approach that works offline, without the need for regular updates even before execution of
the malware. It also shows the dependency for the other products on regular updates, cloud queries
or dynamic analysis".

Webroot is far superior in my opinion and tests like these have no credibility.

 

Which is a "disservice" to other AV products that employ behavior analysis and the like. These products first monitor for abnormal behavior. Once detected, they access the cloud to perform reputational analysis to minimize false positives and user interaction. On the other hand, many AI based products will just alert and leave the decision up to the user.

-EDIT- Also noteworthy is reputation analysis is not exclusively cloud based. SmartScreen for example employs off-line lists that are updated frequently, an hour or less. Many AV vendors likewise have off-line lists they employ.

 

This review is unreliable, very flawed like @vlk has said... If people really want to use CylancePROTECT and believe they are better protected compared to using a normal security product then good for them, but I certainly wouldn't recommend relying on this product.

 

One additional point that needs to be stated. Do not equate vendor endpoint products to their retail ones. They are not equivalent. Endpoint products are designed to be custom configured by internal IT security staff to their particular operating environment. Their "out-of-the-box" configuration per se is minimal security protection at best.