Croatian National CERT test of free antivirus products

http://www.cert.hr/en/start

REPORT original: http://www.cert.hr/sites/default/files/NCERT-LAB-PUBDOC-2011-02-001.pdf

I have translated with google the important captions so you can find the main results.

Figure 3.1 - Graphical representation of percentage of detection
Figure 3.2 - Graphical comparison of the scanning speed in the number of files per minute
Figure 4.1 - Graphical display ratio detection in the second phase
Figure 4.2 - Graphical representation of the tested files per minute (more is better)

It's an on-demand AV test if i'm not wrong.

 

For me two things are particularly interesting. Avira stopped scanning after 10 000 detections, and MSE detection rate was only 0,03% !?

Avira:

" ... If we take into account that Avira did not continue after the detection of 10 000 copies, and BitDefender was markedly slow, one gets the general impression that Comodo, Avast and AVG gave the best results on the test. ..."

MSE:

"... After the scan is completed the tool reported a total of 471,485 files scanned and discovered only 129 copies of malicious code. ..."

I personally believe that something went wrong during the testing.

 

Am I correct assuming that they only looked at the number of scanned / infected objects as reported by the scanner - without mapping them to the actual files in the set?

Funny.

 

exactly , what i was thinking ... I remember norton had problem with this approach as it detected multiple threats as ONE if its same

 

I dont think so, pe, there is a screenshot of avast showing a complete scan of 37775 files finding 9936 malware (files), that means the 26% but them avast score 94%. The same happens with the other av's.

What did you saw to think that?

Zimzi seems to understand the Croatian language maybe he can help us.

 

Yup, that's what they did in the first test.


In the second test, they took 2 x 10 000 files, scanned & removed, then counted the remaining files.

The results of the second test are on page 17.

Tool / Scanned items / Malicious items / Remaining files / Detected files on disk ("real" files, out of those 20k) / Scan time in minutes.

 

Vlk realized what had occurred in the first phase of testing, and dr_Bora explained the second phase. I'll try to sum ​​up the entire test with 2-3 comments which are my personal opinion.

Testing conducted on Windows XP SP3 operating system that was fully updated with the latest patches.

Accurate versions of tested software not listed. Based on testing date (February 1st, 2011) and the images seems that they used: Avast Free Antivirus 5 or 6, BitDefender Free Antivirus 2009, Avira AntiVir Personal 10, AVG Anti-Virus Free Edition 2011, MSE (unknown), Comodo Antivirus (unknown). All antivirus tested with default settings and latest definitions.

All files from the sample collected from a Honeypot. All files are mutually different. Their diversity is confirmed using the MD5 algorithm compression. Each file has a different MD5 summary. 98,83% of all files were .exe, a proportion of those files that do not contain executable code was negligible, below 1%.

The testing was conducted in two phases:

First phase

All files (184 810) are located in the same directory, and the scanning process has started from the Windows Explorer shell environment.

Overall results of the first phase of testing:

Avast - 25,27%
BitDefender - 31,64% (the testing was interrupted due to slow scanning!)
Avira - 47,75% (Avira stopped working after 10,000 viruses detected, so the test was not completed!)
AVG - 29,08%
MSE - 0,03%
Comodo Antivirus - 58,79%

The authors of the test were faced with an "interesting dilema" regarding Avast (as well as others):

"Avast finished scanning of all files in 6 hours 4 minutes and 8 seconds. Reported 170 790 discovered malware,
but it is interesting that Avast reported 675 949 tested files ... The question is how Avast reported over half a million specimens tested. Reply lies at the implementation of anti-virus tools. It is possible that the tool has supports for compressed files that contain more different files. Also, it is possible that Avast scans multiple parts of the same file, and each part separately register as a scanned."

Second phase

The methodology used in the second phase is explained by dr Bora in post #6:

"In the second test, they took 2 x 10 000 files, scanned & removed, then counted the remaining files."

I should add that they used 20 000 files divided into two directories. I did not understand why they were divided into two directories, but it is important that they used 20 000 files at once.

Overall results of the second phase of testing:

Avast - 94,63%
BitDefender - the testing was interrupted due to slow scanning!
Avira - 96,27%
AVG - 94,69%
MSE - 93,06%
Comodo Antivirus - 97,38%

Finally, the conclusion that it is a plain on-demand test. As I wrote, it seems to me that during testing there were some serious problems and failures:

1. Why Avira stopped working during the first phase of the testing after 10 000 viruses detected? I have no malware collection with more than 10 000 malware, so is someone able to check if the problem persists?

2. Is BitDefender really so slow that the first 1000 files scanned longer than two hours as stated in the document?

3. The result that MSE achieved in the first phase of the testing (0,03%) is absurd!? The result achieved in the second phase (93,06%) had to be a clear signal that the first phase of testing took a problem.

4. How Comodo Antivirus could achieve so much better results (especially in the first phase) when (based on personal experience) Avira constantly caught significantly more malware? The answer is perhaps in the fact that during the tests they used default settings of heuristic (medium detection level) and threat categories and that I always use high detection level of Avira's heuristic and all of the threat categories?

 

Avira never stops after more then 10.000 viruses detected. I've done this test several times. BitDefender seems to be slow at scanning in Virtual Machines.

If they used a totally outdated version: BD 2009, then, useless info and test.

This Croatian CERT test is crap.

 

They are testing only free versions (is written in the title): http://www.bitdefender.es/solutions/free.html
the free version of bitdefender is still from 2009 with updated definitions. So is not outdated at all. I would say that BD free is a crap.

In the first phase seems that there was some problems or bugs but the second phase is ok.


@Zimzi, THANKS

 

Avira stops after more then 10.000 on interactive mode

see log (sorry italian version)

no limits in automatic mode.

 

I dont understand this absurd limitation of Avira, the same thing happens in the paid version?