Detection Rates KAV / NOD32 / Anivir / Prevx1

Recently I got hold of a harddisk packed with tons of malware (viruses, trojans, dialers, ...) The guy had obviously no AV-software running and operated XP SP0 (!) with not a single hotfix applied ...

Thought i could use it for some real life AV testing so I came up with the following procedure :

1) Image "bad" physical disk onto a vmware virtual disk ! (only once, for all following test runs)
2) Install and fully update AV on a fresh W2kSP4 virtual machine under VMware 5.5.3
3) shut down machine and mount "bad" virtual disk as 2nd disk (slave)
4) do a snaphot to preserve "bad" virtual disks original state
5) restart machine
6) conduct a full system scan with manually maximized scan settings (scan all files, enable heuristics, unwanted applications ...)
7) repeat test (for verification purposes)

The harddisk had not been in use for at least 2 weeks, so there should be patterns for even the latest piece of malware ! here are my results:

AntiVir PE 7.0.3.............468 detected items ("medium" heuristic setting)
NOD32 2.70.16............1081 detected items
KAV 6.0.1.411..............510 detected items
KAV Rescue Disk.........1303 detected items (offline scan)
Prevx1 2.0.4.19.............30 detected items (extensive scan, full verify, permanently online)

I am aware of the non-scientific character of these tests, but I feel that that two points should be mentioned:

• I scanned and cleaned the disk online using KAV and 510 items were neutralized. A consecutive scan with rescue disk detected another 793 items which sums up to a total of 1303, matching the original detection rate of Rescue disk ! Could that difference be due to the nature of offline scan ? (no open files, no running processes ..) just to remind you: same scan settings and virus definitions in both test runs

• The detection rate of Prevx1 ... I repeated the test just an hour ago with ver. 2.0.5.6 all updates applied, now 34 infections were detected. I already did a thorough research at the prevx thread over at castlecops forum but found no mistakes within my test procedure. After deleting those 34 infections, I mounted the disk under NOD just to see if it's maybe some GUI issue or the way it counts infections and more malware than visually reported has been detected, but NOD found well over 1000 "leftovers" [Edit: 1057 items, to be exact] Is it possible that prevx and the virtual machine interfere in a negative way ??
  I know that Prevx1 has good reviews and is considered a powerful tool but I'd be much more comfortable with a plausible explenation for these strange figures

happy holidays .....

[Post edited several times for clarification purposes]

 

Just curious... You comment that AntiVir's heuristics were set to medium, but make no mention of KAV or NOD32's settings. What were they? Out of the box, neither are at their tightest. Would be more interesting (to me at least), if all three AV's were at their highest (tightest) settings, or even... all three set to their out-of-the-box configurations. Interesting nontheless. Thanks for the post.

Robert

 

6) conduct a full system scan with manually maximized scan settings (scan all files, enable heuristics, unwanted applications ...)



I only mentioned Antivir's heuristic settings, since it's the only configurable heuristics which can be configured in a way ... I thought it would be best to leave it at default settings since preset heuristic detection is always a kind of ompromise between good rates and false positives !

 

I am not surprised by how well Avira detected compared to the others.

 

just for the record, a different "item" count does not necessarily mean that more or less viruses have been detected, this all depends on how the AV reports and counts infections. If AV A triggers on an installer and calls it 1 detection, and then saves the time and does not unpack it because it is known malware, while another AV unpacks it and flags each contained file, you get a different file/item count but the same protection. Same goes i.e. for runtime packed files, does it count the file on disk and the extracted/unpacked/restored temp file as two items or one?

Also, what is an ITEM? A file? A registry key? A system setting? A directory? A tracking cookie? An entry in the host file?

Keep that in mind for such comparisons, item, file or infection count is not a good base for comparisons.

 

That's perfectly true, the numbers can only give you an rough idea of what's going on !

But AFAIK the term "item" relates to a single malicious file (while the term "malicious" still lacks an exact definition ) ... a quick glance at the scanning logs showed no indications of the AVs complaining about settings, HOSTS entries, reg keys ... of course it might have slipped my eyes !

 

That also might be a possible explanation for Prevx1's seemingly horrifying detection rate. I think I remember someone mentioning, that Prevx is no regular AV scanner, but compares files against a common database upon execution !!

It could well be that it doesn't bother to unpack (runtime) packed files for scanning purposes but verifies them on "real" access, so a lot of malware could stay undetected after the first run, but the system is still well protected !
I'll try to figure out how to proove this !

 

yeah some antivirus companys flag detection numbers differently.

dr.web scans my machine clean, then panda says i have 22 infections, what? omg no.... 22 tracking cookies, which ive noticed dr.web doesnt bother with, and am glad it doesnt, they can be a pain to scan and find all these infections.

 

Can someone comment the result of the AntiVir
I'm not very immpresed

 

I would not think of dropping AntiVir as a result of 1 non-professional test ( no disrespect to adam_72)

Otherwise you will be changing your AV every other day!!!!!!!

 

For example if a malware creates 20 files, 5 important and 15 useless without the main program,
Antivirus A report all detected files as a single infection
Antivirus B detects 5 infected files
Antivirus C detects 20 infected files
AntiSpyware D detects 20 infected files + Some other items + 1000 tracking cookies

The 4 programs give the same protection. The problem is determine which detections are actually dangerous.

 

no offense taken

 

Counts are not useful as each AV counts the critters it examines ddifferently, not to mention that each AV might not examine the same files.

An interesting statistic would be how many files were NOT vetted.

 

I have to agree .... the problem lies in the nature of infected "real life disks":
You don't really know the exact quality or quantity of malicious items on the disk, so it's hard to track back how many of them have actually been neutralized in a test run. As long as you don't have a precisely defined set of malware samples (which are reserved to professional institutions for obvious reasons ), these private tests seem to be useless ... and can well cause confusion !

I just realized that and apologize for this "pointless" thread

 

It's never pointless to learn something new!

This thread does have merit to remind folks - I hope - that, as they switch between various products, having a one product flag files not flagged by an earlier product means very little in isolation. It is not immediate proof that a previously installed option "missed" some key malware and that one is now infected, which does seem to be a frequent inference.

Blue

 

Until recently, I had used only NAV.

Never did bother to find out what the counts meant.

Recentky, I used KAV, and the count was quite different, e.g. NAV would claim to have scanned about1.2 million critters, KAV about 2.6 million critters.

KAV does svan archibes that are allegedly not scanned by NAV.

But, there's my big butt again, I created a MSFT Word template that had a single code module.

I do not recall the counts, but NAV clained a much larger number for this single template.

KAV does allow for the creation of a log so you can see what is scanned.
For a system wscan th file is huge, so I wrote a program to list only the items of interest.

 

Once I downloaded 1700 viruses and unpack it, my nod32 catch ALL of tham and 20-30 % of that viruses was unknown, detected heuristically...

no comments..

 

from where did you download those test viri?

 

To all,

This is not a malware trading site. If you wish to be engaged in these types of discussions, take it offline via PM.

Blue