Comparative testing and its weaknesses

izi · Dec 22, 2005

Eugene Kaspersky said:

This part of the article looks at the problems users may have when choosing an antivirus solution. It's assumed that the user will be looking for a product which offers real protection against malicious code. So where can they get information to base their decision on?

The most logical thing is naturally to look at comparative test results from different sources, including professional ones. Do such things exist? Yes, they do, but there aren't many of them. Most IT publications conduct comparative tests of antivirus solutions on a fairly regular basis. They test the solutions thoroughly, and compare everything from the product price to the quality of technical support provided. However, these tests don't really prove the quality of the antivirus function. This is understandable, as testers would need a fairly large virus collection, their own tests stands, and automated testing procedures to thoroughly testing the antivirus component. This means a dedicated group which only tests antivirus solutions, and which requires the necessary resources - something which most IT publications don't have. Comparative tests conducted by IT publications therefore either leave much to be desired, or the publications contact experts who specialize in testing antivirus products.

Currently, the most experienced testers of antivirus products currently are Andreas Marx (Germany http://www.av-test.org) and Andreas Clementi (Austria http://www.av-comparatives.org).These tests describe in detail the quality of detection of various types of malicious programs and the speed at which different antivirus companies react to epidemics. The tests are thorough and detailed, and can be used to compare the characteristics of the antivirus solutions themselves. Sadly, these tests only examine the two characteristics described above; they do not address issues of how antivirus solutions perform in real life situations e.g. when curing an infected system, the reaction of the solution to infected web sites, the amount of resources used, and the thoroughness with which archives and installers are checked.

Sadly, tests which provide an in-depth, accurate picture of how products react in typical situations barely exist. The one exception that we know of is the Test Lab at Moscow State University, which conducts tests using a fairly wide range of situations. However, the methodology of these tests still needs working on, and the university's test lab is not yet known to the public at large.

It's also worth mentioning the tests conducted by VirusBulletin (an industry publication) - I am sure that if I didn't include this, readers would ask why the tests and the resulting VB100% award hadn't been mentioned. Sadly, these tests are far from perfect. The test standards were developed in the mid-1990s and have barely changed since then. Antivirus products are tested using a collection of files infected by ITW viruses. The award is given on the basis of the test results. However, the ITW collection only contains between two to three thousand files - fewer malicious programs than appear in the wild in the space of a single month. Therefore, a VB100% award doesn't necessarily mean that a product really provides protection against all types of malware. It simply means that the product copes well with VirusBulletin's ITW collection, nothing more.

Eugene Kaspersky

Click to expand...

http://www.viruslist.com/en/analysis?pubid=174405517

Log in or Sign up

Comparative testing and its weaknesses

izi Registered Member

Log in or Sign up

Comparative testing and its weaknesses

izi Registered Member

Useful Searches