AV-TEST Advanced Threat Prevention Test Results

offtopic: some prototype exist already. but the cost, safety & security issues are too high to allow their use. Look now all those car accidents while just on the ground, imagine a flying car falling on your 15th floor appartment because a drunk driver... buildings need force shields, until that , no flying car for the public.

btw , skynet is already on development by this famous weird-named tech university, forgot the name...

 

You can now preorder a a roadable aircraft:
https://www.pal-v.com/en/
https://d3m0z8ozms2hlk.cloudfront.net/images/_medium_auto/tuscany-road.jpg?mtime=20170211101806

 

More of the AV-Test thing here: https://www.cylance.com/en_us/blog/real-world-av-testing-with-integrity.html

First, the title of the article, 'Real World AV Testing With Integrity.' You're excused if you go outside and throw up.

Interestingly, Cylance didn't reference the other vendors by name. Looks like the legal issues are starting to catch up with them.

 

I didn't but it made me laugh.
Also test case 4 is missing. I wonder why?
Marketing ploy at it's best.

 

Was never a fan of antimalware testing whether it be av-c or av-test....this is another one of those marketing hypes,they themselves disabled the product features during testing!!

 

yes lol, and since the names aren't mentioned anymore, the whole test because pointless...and will be interpreted as a marketing stunt (which it was anyway)

 

I think all cylance did was repost what av-test did on their site. posted as # 1 in this thread.

this one list more venders than av-test did https://www.nsslabs.com/company/new...anced-endpoint-protection-group-test-results/

https://pages.cylance.com/2017-02-14CNTNSSReport2017_LP-ThankYou.html?aliId=7063965

 

Yeah, I already commented on that here: https://www.wilderssecurity.com/thr...ntion-test-results.392018/page-2#post-2652762

 

the crowdstrike link is page not found..

 

They must have removed it after the court denied their injunction request.

-EDIT- Here it is in another area: https://www.crowdstrike.com/blog/cr...tion-illegally-obtain-access-falcon-software/

 

There is one positive thing that can be attributed to this AV-Test - Cylance debacle. That is the every increasing use of "synthetic" malware that reached "from the sublime to ridiculous" proportions in the AV-Test Cylance test. Also, AV-Test and NSS Labs aren't the only outfits to use it. There is at least one other AV Lab that has been employing in their recent tests; abet not in the absurd proportions done in the recent Cylance tests.

AV Labs are not malware developers. Neither are software security developers. If they were the least bit effective at this, malware would cease to exist. Nor is their any way to independently verify that the "synthetic" malware remotely resembles actual current and future malware development, deployment and execution.

There is one valid use of "synthetic" malware which as I noted previously is not being done by the AV Labs. That is for testing if the security product can be directly bypassed via suspension, termination, or modification by the malware. After all, this is the first thing any decent malware is going to do.

It's up to AMTSO to put a stop to this "crap" as I would put it. If it doesn't, then it can be said that they are indeed nothing more than a "marketing" vehicle for the AV Lab industry.

 

A couple of other "nibbles" I found that hopefully are of interest.

Cylance has employed the "nuclear" option. It has created its own AV testing lab: http://d-4digital.com/msp-launches-av-testing-website-to-stamp-out-misleading-reports/

Comments on that here that I couldn't say better myself: http://itsecurity.co.uk/2016/12/anti-malware-testing-issues/

Personally, when I see such unmitigated aggressive behavior from a commercial concern, I say "run, don't walk away" as fast as you can.

-EDIT-

Definitions

Corporate Suicide - a startup that launches a full frontal assault against a billion dollar plus industry.

Career Suicide - recommending a suicidal startup.

 

Someone just opened a thread on Wilders about Endgame. Endgame appears to be another AI startup concern. Not to intrude on that thread, what caught my eye was that Endgame commissioned SELabs in the U.K. to test and certify it. The methodology was available on-line so I downloaded it: https://selabs.uk/download/cred/20161215002-report.pdf .

At least this test was performed properly in that no comparative analysis was done. What the report did yield was insight into what "synthetic" malware is; namely in this test:

Draw your own conclusions as if this is an adequate way to determine the effectiveness of a security product against 0-day malware. In my opinion, it is not.

 

Endgame -> Nathaniel Fick -> Generation Kill

https://www.youtube.com/watch?v=K9uXLzZyucI

 

I was looking at that the other day. The testmyav site they created has malware you can download if you register and I didn't feel like doing that.

 

Well at least they are with Virustotal now, so I guess we will see how that turns out.

 

Here is why the "special testing" accommodation the AI vendors are requesting "falls flat" so to speak.

Every conventional AV vendor has some unique feature in their software. For example, some employ "generic" signatures that in all likelihood would have detected the modified malware samples used in the SELab's test of Endgame. The AV vendors could and justifiably so based on recent events demand the AV Labs make "synthetic" malware tests for those features. The final result would be total chaos in AV Lab testing resulting in lab test reports becoming useless as any measure of effectiveness.

The fundamental concept of any product test methodology is the establishment of a baseline standard and the measurement of deviation from that baseline.

 

It just seems kind of funny why big names chose the new AI's

Cylance = Dell

Endgame = HP

either they invested it what most here call junk or they know something we don't know.

 

I wouldn't call these products junk.

I would classify them for what they really are - an intelligent host intrusion protection system(HIPS). Used as supplemental security protection, I believe they have great potential. The "myth" being perpetrated is that by itself, it is adequate security protection.

The problem is the established AV vendors already have or are in active development of their own AI solutions. Symantec is employing the new advanced AI algorithms in its Sonar product. Sophos has acquired Invincea, etc.. This leaves the startups no where to go but to try to "go it alone." The problem is a disastrous marketing strategy that targeted the enterprise endpoint market rather than going after "the low hanging fruit" i.e. SMBs and the retail market.

-EDIT-
As mentioned in the Cylance thread previously, Dell is using Cylance on their thin-clients. I suspect HP is using Endgame in the same way.

Definition of a "thin client" here: https://en.wikipedia.org/wiki/Thin_client

 

Eset has a blog posting, a portion of which relevant to these recent testing "accommodations" done by AV Labs, I have extract below. Again, it's up to AMTSO to clean up the "mess."

Ref.: http://www.welivesecurity.com/2017/02/13/next-gen-security-software-myths-marketing/​

 

Haven't read everything, a bit too much info. But from what I understood competing software companies disagree with the testing methodologies? What else is new, often sponsored tests are not really that trustworthy. But anyway, if Cylance really is that good, why don't they release a consumer version.

 

Like I posted previously, they aren't interested in the "low hanging fruit" i.e. retail market. They want the "whole enchilada" i.e. desktop endpoint market.

 

Cylance doesn't but they work with somebody that does. Malware Managed https://www.malwaremanaged.com/

They had a rep that came here but I am guessing Wilders member scared him away.

 

Malware Managed is not a consumer centric company, they're a small business/enterprise reseller and they provide AV management. Basically they're outsourced IT.

 

Here's a tweet: https://twitter.com/da_667/status/831350334912069636 that in somewhat corse and simplistic language describes what NSS Labs does.

Unlike the smaller conventional AV Labs, NSS labs does penetration testing. In essence, they create test scenarios that purposely try to exploit commercial security appliances and software. Many of these scenarios are not based on real world malware, but are artificial tests specifically created to find OS and app software and hardware vulnerabilities resulting in the bypass of security hardware and software.

I classify penetration and malware effectiveness testing as two separate and distinct procedures. Elaborate and expensive penetration testing is a must for large corporate IT infrastructures.

On the consumer retail security software side, the only one I know of that does like testing is Rubenking over at PC Magazine using SecTools Core Impact software which BTW costs $30,000. Very simplistic penetration user tests are the old Comodo and Matousec leak tests.