Interesting ShawdowServer results

Without risking the A is better than B scenario, I found the latest ShadowServer zero-day test results very interesting. There seems to be (in terms of the test) a large spread between the top and bottom ranked AVs.

http://www.shadowserver.org/wiki/pmwiki.php?n=Stats.VirusDailyStats

http://www.shadowserver.org/wiki/pmwiki.php?n=Stats.VirusWeeklyStats

 

id love to know what they have changed.

for the past 12+ months, drweb scored 90+ every single day.

yet, for the last 3-4 weeks, drweb scores 2-8%.

ive noticed these changes, since the addition of other AV's too.

id love to hear some 'official' comments from them about it.

 

Wow, such a huge difference between avast home and pro.

 

A: OMG MY AV ROX AND YOUR AV SUXORS
B: NO WAY THEY TEST CRAP AND EVERYTHING IS FULLY BIASED

Just thought I could get the fan boy stuff out of the way before we start having a real discussion

It is such a huge discrepancy and I cant even explain it. The last time I looked Avast wasn't doing so bad so I thought it must have been a recent 'hit' they took. But I looked at the yearly stats and they are doing just as bad but I clearly remember they were never ranked so low. Maybe there is a mistake?

 

Clam-AV is where it should most likely be, but avast is under it, so i am confused with this, by my understanding Avast and Avast pro use almost exactly the same engine.

 

In terms of detection avast! Home and Professional are exactly the same, I don't believe the results...

 

Regarding avast!, see this thread:
http://forum.avast.com/index.php?topic=36109.0

Moderator states:

-----

So if the detection capabilities are the same, avast! home should rank the same as the professional product. Shadowserver however, are using the linux versions of the product (which aren't as high in detection rates). See: http://www.shadowserver.org/wiki/pmwiki.php?n=Stats.Viruses

Also, Shadowserver are using the avast! linux home edition version 1.0.8, not the latest version 4 or the windows version 4.8 most are using.

The Kasperksy version shadowserver is using, is Kaspersky Anti-Virus for File Server, version 5.5.18.

 

I don't believe those results. Different from any possible experience I had and any av-comparatives/test I 've seen.

 

Personally, I've always disliked ShadowServer's statistics and think they have always been a very misleading representation of detection rates of AVs... my opinion still stands

 

Why are they misleading? What is wrong with how they are doing it or how they are reporting it?

 

I would take their results with a pinch of salt.

For example, VirusBlokAda have had a long battle with them in trying to get them to test VBA32 fairly. They were testing VBA in demo mode as their license had expired!!!!!! Recent results suggest that maybe, just maybe they now know how to use VBA32 (and other AVs)!

And overall Honeypots are well-known to contain a high proportion of corrupted samples.

I agree; the use of Linux versions and honeypots do not help but it is also their methodology that I question.

 

Good question.

It may be more of the mantra here that if your AV tested well the methods are scientific and statistically valid... if it did not test well, the methodology is obviously flawed...actually I have no idea if the results are valid or not.

 

What put me off shadowserver is the fact that certain products....using the same engine...were getting different results for the same detection. Now if that is not an indication that something is amiss I'm not sure what is

 

Exactly... if there was a consistent setup of AVs, certain AVs which share the same engine such as Kaspersky and F-secure will have the same number of detections. In ShaddowServer's case, a signature detection (for example of Trojan.Win32.VB.bzr), Kaspersky gets 32,000+, whereas F-Secure gets a little below 25,000... if its the same signature, why is there a discrepancy; must be from inconsistant settings.

Also, as Blackcat rightfully mentioned, the use of honeypots results in many corrupted samples (samples which do not and can not infect you) and honeypots do not necessarily catch samples which normal users may be infected by. Also tests are carried out using Linux versions of AVs, whereby results can vary against Windows versions (which most readers use) due to ability to unpack etc.

 

Not with me... We all know the AV industry is a competitive and dynamic one and its not easy to be the best because all the other AVs are trying to reach the same goal... everyone cant be the best! The AV I use does not reach 100% or the highest in all tests, but I dont mind, I know it doesn't detect 100% of samples and I know which AVs detect more than mine... if another AV detects well, good for it

 

cant argue your thoughts dawgg.

 

They don't share ths same engine. F-Secure uses the old Kaspersky engine.

 

they are both linux versions, so kav for linux is still 5.5, there's no heuristics engine like in the windows version 7 and so on.

 

Detection rates should still be the same between the versions used in ShadowServer (because it uses a Kaspersky version which has the old engine, which F-Secure also uses). Signature detection of Kaspersky should be the same as F-secure in ShadowServer, the only detection difference in the versions ShadowServer uses should be that F-Secure also has its other engines, but in the results, F-Secure's engines dont detect anything (possibly another flaw of using older linux versions; current Windows version of F-Secure may be able to detect more using Heuristics). In this case, there is no evidence any of the heuristics were used, so F-Secure and Kaspersky are only getting detections using Kaspersky's engine, so the results should be the same. Kaspersky didnt have its Heuristic module in the version ShadowServer uses which is why the results are partially flawed; in reality, most users will have heuristics detecting malware as well in Kaspersky).

Other AV products will also most likely have discrepancies as such and have current windows versions of their products detecting far more than ShadowServer does. Eg, DrWeb's Windows home user products uses the Orion heuristics... ShadowServer doesn't, hence in the real world, Orion may detect malware, but ShadowServer will show it does not detect it.



I am not attacking ShadowServer and it is good that it is transparent with its setup and information about its methodology, but the problem is that many people only look at the numbers and know absolutely nothing about the outdated versions compared, samples (many of which are likely corrupted) and OS; hence in the real world, its not showing a proper up to date comparison of detection capabilities; hence IMO, ShadowServer's results should not be considered when comparing detection capabilities of AVs for home-users (most of which are Windows users).
Also, seeing as it shows one signature is capable of detecting far over 50% of the malware detected that day, it does imply the samples used are questionable to say the least.

 

I always wonder why Symantec is not represented. Do they not make a Linux version or is it too expensive?

 

norman, f-prot, avast, avg, nod32, mcafee & vba32...all of them above kav...oh well

 

IMHO there is no way to disable origin detection in 4.44 engine.
So, ShawdowServer still use 4.33 version?

 

this site deals with zero day threats, so if you look hard enough, there is a lot of reality to their findings.

 

But that "reality" is not acceptable to many of the true believers whose AV is in the lower end of the rankings- including Kaspersky, etc.

 

No, that "reality" is not acceptable when F secure is getting twice as many detections as KAV using a Kaspersky detection from the Kaspersky engine.


That and the fact that most honeypots are filled to the gills with corrupted samples that most vendors will not add to their detection lists. I don't see any methodology there that states they check for corrupted samples, the frequency of updates and what settings each engine is tweaked to.