VirusInfo Test Results

Hi All,
I see the older thread was for July 2008 results, so I am starting a new generic one for VirusInfo Test Results.

How VirusInfo tests:
The testing of anti-viruses by VirusInfo is powered by free on line scanner VirusTotal. Project participants, being practicing specialists in the area of computer security, are uploading at VirusTotal the malicious software that they have received from infected machines, and then publish the results of scanning in a special topic on VirusInfo forum. The malicious software should meet the following requirements:

1) The sample should not be detected by the anti-virus software that protects the infected machine.

2) The sample should be found by the consultant him/herself in a real infection case.

3) The sample should not be taken from some other site or from some other collection of malware

Results:



Also available at http://virusinfo.info/index.php?page=testseng

Avira and Webwasher lead the pack. With GData, AVG, BD, Ikarus and MS putting as strong showing.

 

wow
Microsoft is doing very well

 

Virustotal results. Not really reliable.

 

I agree, we have checked thousands of files coming from Virustotal and quite a big portion of them was corrupt and non-functional. However, this does not imply that every undetected sample is corrupt, but many of them actually are as we have analysed them and can confirm that.

 

very nice performance for my favorite AVG. good sign for microsoft they are improving.

bad performance for nod32, kaspersky, norton and trendmicro

 

"WebWasher uses the Avira engine and a self-developed heuristic engine"
http://www.virusbtn.com/news/2008/09_02.xml
But it seems like WebWasher's heuristic engine is not in use at this Virusinfo test.

The result of Ikarus (only signatures) seems strange, but I have tested the virus.utilities recently and there were only very few heuristic detections, even with the program.
But besides that, it detected nearly every packed file as 'Trojan-...' or 'Packer...', no matter if it was real malware or not.
Seems to be enough for a good result at Virusinfo.

Cheers

 

I can't comment on the reliability of VirusTotal (since I have no knowledge of the same).
But from what I have seen in the VirusInfo forum, it appears they have a good set of security volunteers. Who detect many real world malware. So unlike other tests, I tend to think of VirusInfo samples to be of higher quality.
But does VirusTotal actually scan / upload files properly Don't know. Relevant experts please comment.
Also a point to see is the sample size of VirusInfo. Its almost always less than 50, due to their strict requirements (as stated in my starting post). So while most VirusTotal samples may be bad, these are all but a small subset of valid ones.

IMO, its best to take VirusInfo,AV-comparative & AV-test results in comparison. To effectively, rate a product. A good product like Avira will ace at least 2 out of 3. Some products may do good with a certain test but not in others. I use this to decide.

@Subset:
Thanks for the info, I was aware that WebWashers uses Avira. But not of their in-house heuristic engine.
I think Ikarus and Esafe are in same class. I have used Esafe equipment, it also marks nearly every exe or packed file as suspicious.
Since VirusInfo doesn't check for FPs or Removal, such products will seemingly have an edge.

 

Well..

Nod32, Kaspersky, F-secure, McAfee, Panda, eSafe, F-Prot, Trend, Symantec, and Norman scored much better than Dr.Web in recent av-test.org, but equal or worse than Dr.Web in this virus info test. Dr. has also been leading at shadowserver.org for few weeks now.

really difficult to rate drw if one should pick several tests and try to find a pattern for it

 

Like the Eset guy said. I've had countless samples uploaded to virustotal where my AV Kaspersky said it wasn't infected. I got scared and worried. And I send it to Kaspersky and other av vendors if I'm feeling nice just to find out the file is corrupt and therefore harmless so it needs no detection. It happens. A lot.

 

drweb usually scores well here, but still a flawed and useless test.

i have my own forms of testing that i run every product i use or try through, my own ways, only then do i know what each can do.

better this, than to rely on all these flawed tests.

this usually includes going to some known bad websites, known bad downloads and p2p all through VM, put this along with pc benchmark tests that i usually do and take factors such as support, price etc with all that, my choices are usually made, at least... this is how it was when selecting drweb all those years ago.

as for SS, not sure what i think about them either, i certainly wont be on the bandwagon agreeing with them because drweb scores highly there, & they certainly do go through a **** load of samples per day.

i prefer to make my own mind up about things, instead of being another Pwn in society.

 

Yep, but Dr.Web should do a lot worse or those others should do a lot better. I've understood from the comments of Dr.Web employees that they don't add garbage files at all and there are actual people analyzing/creating signatures instead of some automated system. Atleast they've blamed that there are lots of junk files in some tests, was it av-comparatives or some other that they had argument about this some time ago.

 

Well the easiest way is to take the cumaltive score off all the tests
But really best if you pick a couple of test which you think are valid and neutral. Then take their effective score.

By the way, this is an open question to everyone. I just want to know, for my own knowledge.
If VirusTotal corrupts samples, then how come some AV products detect it ?? Say if Sample A is detected by 4 scanners, it can't be corrupt right ? If yes, then VirusInfo has checked with a minimum of 35 valid samples, from which (just for example) Clam missed 29.

 

Maybe to increase their detection rate in tests like this?

 

I don't think virus total corrupts anything, but the files are corrupted before uploading.

Here's my good old trusty example: http://risl.codename.fi/example_jotti.JPG

If some reputable av like Kaspersky etc. add a signature for a file and every AV vendors participating can receive the results and samples, I strongly believe that these smaller and less known vendors simply add automatically a signature for that file without even taking a closer look at it. Same detection names? Come on!

It probably isn't that hard to create a program that simply grabs some amount of bytes from the received sample(probably by dumping from memory)and then creates the signature(string)what to look for. Of course to make it more obvious: don't even bother to use the same detection name.

Easy way for some vendors to success at these "tests". ?

 

Ow I've seen that behaviour very often. Especially by Ikarus.

 

You all seem to worry far too much about these test results,its how an AV works in the "real world" that matters,like an earlier poster eluded to if you have a product that detects almost all types of packers,corrupt files etc as trojans/malware its detection rate is going to be very high,not accurate but high all the same and would look good in a detection test compared to other more accurately detecting products,but guess which would be the best product to have installed on your PC?

 

The art of recycling, you will find detection names from most AV vendors.
But if you scan malware samples, Ikarus has a pretty high detection rate and currently very few false positives (packed files excepted). Meat and potatoes for most users.

Cheers

 

It's also been stated for weeks that SOME av's are having issues with shadowserver.org with the parsing of logs along with samples that are detected being logged as not detected also having simple issues with Shadowserver saying they sent such and such Sample and the vendor not receiving such and such sample.

as for hard to rate.. well knock out Shadowserver and Virus total.. both which people seem to think have Questionable results

And use Av-org and AV-Corp along with VB (which they just dropped out of ) shows a very decent pattern. I would think

If web would spend more time working on there product and less time trying to discredit tests maybe V5 would be out before 2010. maybe then they would not have to drop out of tests

Edit.
Just a FYI on shadowserver. I'm only going off what I have read here on wilders and on the general internet. I have not done any serious looking into it my self.

 

Well, I see Avira & WebWasher detected 35 samples. Hence we know for sure that 35 samples didn't suffer from corruption.
Now out of the 35 proven valid samples, ClamAV detects only 6. Similarly for other apps.

And with regard to the VirusInfo sampleset, IMO its the most real-world sample than any other test. Its samples have too pass these criterias:
1) The sample should not be detected by the anti-virus software that protects the infected machine.
2) The sample should be found by the consultant him/herself in a real infection case.
3) The sample should not be taken from some other site or from some other collection of malware.

 

if you were gonna do that, then all of them should be knocked off as ive got problems with them all.

creating new technology takes time, the same does not apply to companys that dont have any.

and FYI, drweb have pulled out of tests because drweb have found flaws in them, "a flawed test, is an invalid one!"

whether avira are having problems with their score on their or not, still does not explain that currently, drweb beats em all.

1.3 million samples tested in just 1 month is a big enough test id say, so its just sooo easy to discredit any tests you dont like.

currently its Avira complaining about their test scores on there, however... for 6 months, they were testing 4.33 even though 4.44 with its greater detection was out. Also, during the switch over to 4.44, for a month drweb scored 1-3% due to them doing it wrong, so kinda levels the playing field, dont you think.

 

If a sample is corrupted and then how can it still be detected ?
And even if its magically detected, then it means the AV product has very good signature,heuristics and scanning capability IMO.

 

Virustotal scanners are first of all not the latest versions for the programs they represent: Symantec is version 10 while the product is 15.
Also you don't have any assurance they are using the maximum settings and also many corrupt samples are detected as infected by 3-4 antiviruses and that doesn't mean the file is dangerous.

So, this test is not too reliable... maybe only 85 % reiability.

 

So Pykko, If I understand correctly even if a sample is corrupted, it still causes detection by some scanners of VirusTotal.
But from what I have seen via VirusTotal forums it seems, that the analyst who submits the sample also cross-checks that the in-the-wild sample is correctly detected.
So with all probablility in place I think your estimation of 85% would be correct

 

In your posts you have always had problems with them. Anyone where web has scored low it has been unfair advantage. or something sim.

As stated in my post it says this is just what I have read. also it says what tests that people seem to question a lot. this is not me complaining about tests not at all this is just what I have seen here and on the web. do a forum search. if you don't believe what im saying.

How much time.. that's all I got to say.. and its not new Tech its the same thing the other company's have Web is just finely getting with the times.

Exactly. to much time spent worrying about tests. but according to you all tests are "flawed" unless they score in your favor. we could spend all day going round and round about this. Lets not.

Yes. it seems to be aint it.

Will you get off Avira I'm not even comparing them.. I'm simply saying that from what I have read in this forum shadowserver is having some issues with SOME AV's. Did you ever think maybe web was included.

 

Just a Curious question if anyone knows.. Is the Sunbelt result in the list Viper ? or a different AV ?