some big names fail the latest VB certification

Some very interesting reading concerning the latest VB certification=== "Security giants fail Virus Bulletin test"


http://www.vnunet.com/vnunet/news/2227328/vb100-takes-windows-server

 

Failed for 1 false positive! That's terrible! Let's all and change our avs!

 

Passed:
Agnitum,AhnLab,Alwil,AVG,Eset,Fortinet,FRISK,Kingsoft,McAfee,Microsoft Forefront,Norman,Rising,Sophos,Symantec,AEC (Trustport) and VirusBuster.

Failed:
ArcaBit,Avira,CA eTrust,F-Secure,Kaspersky,MicroWorld,Quick Heal and Redstone.

 

To fail with 93 misses (Arcabit ArcaVir) is one thing, to fail with 1 false positive (all of the others except for CA eTrust 1 miss, Redstone 1 miss + 1 false positive) makes the whole test a bit of a joke the way they word their results.

 

VB looses its credibility for good.

 

Why? False positives can be a major problem.
It's only one test. Nothing more, nothing less.
The report allows companies to fix their false positives.
This is a good thing.

 

Nobody is saying that FP tests should be abolished. Osaban's point is that to declare a product as having "failed" the test just simply because there was a false positive is ridiculous, even more so when we all know the products that "passed" the test certainly do not have 0 FPs in real life.

Yes, and the value of tests is based on whether they can provide an accurate and/or meaningful gauge of the tested products' abilities. VB100% cannot.

 

Originally Posted by saberfox:

This criteria has been in place for many years, unlike other tests that select arbitrary cut offs for ratings.

Originally Posted by saberfox:

The same could be argued for any test. That's why many AVs have removed themselves from testing.

 

Many? No only those that were in the bottom of detection and only in those test they failed, not the ones where they shine.

 

Yes, it's been for many years: since the late 80s, if I remember correctly. Given how quickly the malware landscape is changing, you're doing nothing but kidding yourself if you think a metric from 20 years ago is still useful today.

This thread is about VB100%, not other tests. If you want to bash other tests, feel free to create your own thread.

 

Originally Posted by saberfox:

.
Speak to the folks at VB100 to change their criteria.

 

"Many," you say?

Hmmm... 10 would qualify as "many". Please name at least 10 AVs that have removed themselves from testing.

 

The VB100 test is what it is: Pass or Fail the test as set.
Not getting all the mals is bad, very bad. The FPs are bad too.
The issue, obviously, is the statistical problem of the tiny sample.
(although I dont know the sample size of the 'clean' sample)
Ahh, we have all come to live with and love metrics...

Whatever the results, the VB100 organisation has support from the vendors, even Trend who have withdrawn from the test attend their conferences.

And as we know a group of tech ubergeeks can really live the high life
http://www.sophos.com/blogs/gc/g/2008/10/03/vb2008-lipstick-pigs-anti-virus-and-pony-tails/


PS Anyone want to submit a collective adjective for that group

 

I think Shaun Nicols catchy headline is the problem here. "Failed" is a bit strong.

But journos have gotta eat too.

 

Agreed this is only one test, nothing more, nothing less, and VB100 is a very respectful organization. Still the perception that one has about the performance of every AV is flawed: In the August AV Comparatives test as an example, Avira's detection was the best with 99,2% (failed VB100) compared to McAfee's 84,4%(passed VB100).

Avira which is the best AV in detection, at the moment, wasn't even mentioned as a significant company in the article 'Security giants fail Virus Bulletin test' thanking VB100 "antiquated" services.

 

Yes.
A problem re credibility isn't it.

 

Eh, these tests are apparently a piece of junk not reflecting real world at all ...