2006-03 AV-test from Jotti's 6 x 100 snapshots

[Firefighter]

Hi again. Because so many av:s had upgraded their engines since last fall, I made a new Jotti's av-test against snapshots. The brand new Jotti's av-test with the new specifications, there have to be AT LEAST TWO detections to minimize those False Alarms!

2006-03 AV-test from Jotti's 6 x 100 snapshots:

All samples in these snapshots are NOT with "ZIP", "RAR", "CAB" format or those "COM" format, that are actually old DOS samples. (most of all are with "EXE" format).

Those snapshots were excluded, where all av:s were capable to detect the sample, but there have still to be at least two detectings in each snapshot.

Checked as viruses/worms:

Total ------ Set 1 ------ Set 2 ----- Set 3 ----- Set 4 ---- Set 5 ----- Set 6

_6.3 % ----- _6 % ----- _4 % ----- 12 % ----- _3 % ----- _8 % ----- _5 %

================================================================================

Detection rate:

Total ------ Set 1 ------ Set 2 ----- Set 3 ----- Set 4 ---- Set 5 ----- Set 6


72.3 % ----- 82 % ----- 69 % ----- 78 % ----- 72 % ----- 61 % ----- 72 % -- DrWeb 4.33
72.2 % ----- 78 % ----- 76 % ----- 64 % ----- 71 % ----- 74 % ----- 70 % -- Kaspersky
70.3 % ----- 70 % ----- 71 % ----- 66 % ----- 74 % ----- 68 % ----- 73 % -- Vba32
62.3 % ----- 62 % ----- 57 % ----- 59 % ----- 74 % ----- 63 % ----- 59 % -- AntiVir
59.0 % ----- 71 % ----- 63 % ----- 52 % ----- 54 % ----- 60 % ----- 54 % -- NOD32
58.3 % ----- 65 % ----- 63 % ----- 55 % ----- 58 % ----- 55 % ----- 54 % -- BitDefender
46.0 % ----- 55 % ----- 43 % ----- 38 % ----- 49 % ----- 49 % ----- 42 % -- Fortinet
45.3 % ----- 51 % ----- 47 % ----- 43 % ----- 43 % ----- 52 % ----- 36 % -- Norman VC
43.2 % ----- 49 % ----- 46 % ----- 33 % ----- 55 % ----- 41 % ----- 35 % -- ArcaVir
42.5 % ----- 53 % ----- 50 % ----- 39 % ----- 42 % ----- 41 % ----- 30 % -- AVG
32.8 % ----- 40 % ----- 35 % ----- 36 % ----- 35 % ----- 25 % ----- 26 % -- Avast
28.0 % ----- 26 % ----- 28 % ----- 32 % ----- 34 % ----- 30 % ----- 18 % -- ClamAV
24.3 % ----- 37 % ----- 27 % ----- 23 % ----- 24 % ----- 23 % ----- 12 % -- F-Prot
24.0 % ----- 35 % ----- 20 % ----- 20 % ----- 31 % ----- 17 % ----- 21 % -- UNA
17.0 % ----- 26 % ----- 19 % ----- 21 % ----- 16 % ----- 12 % ----- 08 % -- VirusBuster

================================================================================

Here are those ProActive like detections:

ProActive (heuristics + behaves like + based + BACKDOOR.Trojan + DLOADER.Trojan+ DLOADER.IRC.Trojan + GenPack: + MULDROP.Trojan + STPAGE.Trojan + Win32:Trojan-gen + WIN.IRC.WORM.Virus + Crack + fam/family + gen/generic + modified + probably + variant etc.) detection:

Total ------ Set 1 ------ Set 2 ----- Set 3 ----- Set 4 ---- Set 5 ----- Set 6

31.7 % ----- 27 % ----- 38 % ----- 32 % ----- 31 % ----- 36 % ----- 26 % -- NOD32
20.5 % ----- 27 % ----- 27 % ----- 14 % ----- 22 % ----- 20 % ----- 13 % -- AVG
15.3 % ----- 10 % ----- 23 % ----- 19 % ----- _8 % ----- 13 % ----- 19 % -- BitDefender
14.0 % ----- 18 % ----- 21 % ----- _8 % ----- 15 % ----- 10 % ----- 12 % -- Avast
13.3 % ----- 17 % ----- 14 % ----- 18 % ----- _5 % ----- 12 % ----- 14 % -- DrWeb 4.33
_9.7 % ----- _0 % ----- _7 % ----- 12 % ----- 14 % ----- 14 % ----- 11 % -- AntiVir
_9.0 % ----- _6 % ----- 10 % ----- 10 % ----- _9 % ----- 12 % ----- _7 % -- Norman VC
_7.0 % ----- _8 % ----- 12 % ----- 11 % ----- _3 % ----- _2 % ----- _6 % -- Vba32
_6.0 % ----- _7 % ----- _8 % ----- _3 % ----- _9 % ----- _5 % ----- _4 % -- F-Prot
_4.8 % ----- _5 % ----- _7 % ----- _6 % ----- _5 % ----- _4 % ----- _2 % -- ArcaVir
_3.5 % ----- _2 % ----- _3 % ----- _4 % ----- _3 % ----- _4 % ----- _5 % -- Fortinet
_3.2 % ----- _4 % ----- _6 % ----- _5 % ----- _2 % ----- _2 % ----- _0 % -- VirusBuster
_2.8 % ----- _6 % ----- _2 % ----- _3 % ----- _3 % ----- _3 % ----- _0 % -- Kaspersky
_2.2 % ----- _1 % ----- _3 % ----- _4 % ----- _1 % ----- _2 % ----- _2 % -- ClamAV
_0.2 % ----- _1 % ----- _0 % ----- _0 % ----- _0 % ----- _0 % ----- _0 % -- UNA

================================================================================

Here are those ProActive like detections from those files MISSED BY SIGNATURE:

43.6 % -- NOD32
32.5 % -- DrWeb 4.33
26.9 % -- BitDefender
26.3 % -- AVG
20.4 % -- AntiVir
19.1 % -- Vba32
17.2 % -- Avast
14.1 % -- Norman VC
_9.3 % -- Kaspersky
_7.8 % -- ArcaVir
_7.3 % -- F-Prot
_6.1 % -- Fortinet
_3.7 % -- VirusBuster
_2.9 % -- ClamAV
_0.2 % -- UNA

(At least ArcaVir and maybe UNA, were not capable to use their (best) heuristics in Jotti's)


Best regards,
Firefighter!

[Marcos]

Take it always with a grain of salt, a lot of files uploaded to Jotti are either installers or corrupted files. If these were filtered out, NOD32 would have much better detection rate :-) Re. installers, I found about 1000 pieces in less than a week that would be detected after being unpacked.

[Happy Bytes]

I just give one example here - and just for the note - everyone who flags this file is producing a false positive because this file is innocent, i just checked it after Marcos did send it to me:

> "winupacke.exe" file.

> Antivirus Version Update Result
> AntiVir 6.34.0.53 03.18.2006 Worm/Mytob.BT
> Avast 4.6.695.0 03.17.2006 no virus found
> AVG 718 03.17.2006 no virus found
> Avira 6.34.0.53 03.18.2006 Worm/Mytob.BT
> BitDefender 7.2 03.18.2006 no virus found
> CAT-QuickHeal 8.00 03.18.2006 (Suspicious) - DNAScan
> ClamAV devel-20060126 03.17.2006 Worm.Mytob.Gen-6
> DrWeb 4.33 03.18.2006 no virus found
> eTrust-InoculateIT 23.71.105 03.18.2006 no virus found
> eTrust-Vet 12.4.2123 03.17.2006 no virus found
> Ewido 3.5 03.18.2006 Worm.Mytob.bt
> Fortinet 2.71.0.0 03.18.2006 W32/MyTob.BT!net
> F-Prot 3.16c 03.17.2006 no virus found
> Ikarus 0.2.59.0 03.17.2006 no virus found
> Kaspersky 4.0.2.24 03.18.2006 Net-Worm.Win32.Mytob.bt
> McAfee 4721 03.17.2006 no virus found
> NOD32v2 1.1449 03.17.2006 no virus found
> Norman 5.70.10 03.17.2006 no virus found
> Panda 9.0.0.4 03.17.2006 no virus found
> Sophos 4.03.0 03.17.2006 no virus found
> Symantec 8.0 03.18.2006 no virus found
> TheHacker 5.9.5.115 03.17.2006 W32/Mytob.bt
> UNA 1.83 03.16.2006 no virus found
> VBA32 3.10.5 03.17.2006 Net-Worm.Win32.Mytob.bt


And now the question is how much would you trust sentences like "The samples where detected by at least 2 different scanners".

ClamAV is producing this false positive because they created a signature over the Upack unpack stub. (Worst Chase)
QuickHeal (DNA-Scan) flags even a wet poop if it's runtime compressed
Kaspersky is the initiator of this Signature False Positive and the rest (that's another sad story) only includes it because Kaspersky detects it. If someone would have analyzed this file in a proper way they would have noticed that it's NOT malware.

[Stefan Kurtzhals]

This is the original UPACK compression tool, not malware.

[Happy Bytes]

-{ Quote: "This is the original UPACK compression tool, not malware." }-

Thanks for confirming this Stefan ;)

[JimIT]

-{ Quote: "I just give one example here - and just for the note - everyone who flags this file is producing a false positive because this file is innocent, i just checked it after Marcos did send it to me:

And now the question is how much would you trust sentences like "The samples where detected by at least 2 different scanners".

If someone would have analyzed this file in a proper way they would have noticed that it's NOT malware." }-


SHHH! What are you trying to do?? Impart reason to this discussion?? We need to make sure our antivirus is DOING SOMETHING!! :D ::) ;)

It MUST be a virus, because XYZ Antivirus SAID SO!! ::)

[hemkop]

Thanks for confirming this JimIT :P
It is a virus right :P ghaghaghagha

[Happy Bytes]

-{ Quote: "Thanks for confirming this JimIT :P
It is a virus right :P ghaghaghagha" }-

Sm0kie, can you play somewhere else the troll please? ::)

[Firefighter]

-{ Quote: "
And now the question is how much would you trust sentences like "The samples where detected by at least 2 different scanners"." }-I made this limitation just because certain scanners were probably detecting too many FP:s.

Best regards,
Firefighter!

[pykko]

Well, like Marcos said many files are not detected because not all AVs support every type of malware, and there are many FPs. I know from my experience...files detected by KAV, VBA32, Bit Defender which were only corrupted files, or incomplete virus bodies. BD even detected the virus code copied in a HTML file as infected file. ::)

[Firefighter]

-{ Quote: "Well, like Marcos said many files are not detected because not all AVs support every type of malware, and there are many FPs. I know from my experience...files detected by KAV, VBA32, Bit Defender which were only corrupted files, or incomplete virus bodies. BD even detected the virus code copied in a HTML file as infected file. ::)" }-After all, I think that ALL scanners are making FP:s sometimes, that's why at least two detectings too.

Best regards,
Firefighter!

[Firefighter]

-{ Quote: "Sm0kie, can you play somewhere else the troll please? ::)" }-Is SmOkie really a Swedish? Jag tror at de är bättre att jag kan sända privat besked till honom/ henne på svenska! ;D

Best regards,
Firefighter!

[Firefighter]

-{ Quote: "I just give one example here - and just for the note - everyone who flags this file is producing a false positive because this file is innocent, i just checked it after Marcos did send it to me:" }-Are you trying to say that NOD's heuristics is more accurate that the signature detectings of any other av:s? Why this, NOD was the only that was awesome in heuristics in this test.

Best regards,
Firefighter!

[Devil's Advocate]

-{ Quote: "SHHH! What are you trying to do?? Impart reason to this discussion?? We need to make sure our antivirus is DOING SOMETHING!! :D ::) ;)

It MUST be a virus, because XYZ Antivirus SAID SO!! ::)" }-

Damn newbies. Everyone knows you need detection by at least two antiviruses, 'XYZ and ABC' to rule out false positives. :)

[pykko]

-{ Quote: "Damn newbies. Everyone knows you need detection by at least two antiviruses, 'XYZ and ABC' to rule out false positives. :)" }-

Well, here's 7 AVs detecting a corrupted file as virus.
Even if it's infected the file is damaged and can't be excuted. ;) So you can't say 4, 5, 9, AVs detected this and it'a certainly a virus. ;D

[Stefan Kurtzhals]

Why, it was a malware sample, only damaged. Those cut-off corruptions often happen during mail or ftp transfer (exploit). And it is actually ok to filter such files at the gateway.

So detection of it is ok. Those AV programs who fail to detect it rely on a signature after unpacking the sample or the part of the file which contains the signature is cut off. Unpacking doesn't work anymore because the file is corrupted.

[Tweakie]

Another "real world" interesting test is the one performed by the nepenthes development team. It's a bit outdated now (Dec. 2005) but I did not see it mentionned on this forum.

The intersting thing is that this test has been performed on malware captured in the wild by nepenthes sensors (and not from collector's websites such as vxh**vens) . The test has been performed on a Linux platform and it looks like not all available options (heuristics, etc) have been activated for every scanner (they do provide the command line used). Keep this in mind when looking at the results.

As a snapshot of current malware activity, it may better reflect the real world performance of an antivirus gateway than tests that include either a huge collection of "old" malware or only new/unknown malware.

The test is available here :
http://nepenthes.mwcollect.org/stats:scannertest

The summary table :

1 Antivir 99,04% +7,07%
2 BitDefender 96,23% +1,52%
3 VirusBlockAde 95,17% +1,42%
4 F-Prot 94,02% +2,39%
4 Authentium 94,02% new
5 Norman Virus Control 93,78% +1,19%
6 Fortinet 87,29% +2,35%
7 F-Secure Antivirus 85,22% +5,99%
8 Kaspersky 85,10% +5,73%
9 VirusBuster 82,53% +11,76%
10 Trend Micro 76,19% +5,14%
11 ClamAV 71,41% -0,85%
12 NOD32 70,06% +4,05%
13 Sophos SWEEP 68,58% +2,45%
14 eTrust 63,97% new

[Ned Slider]

-{ Quote: "Another "real world" interesting test is the one performed by the nepenthes development team. It's a bit outdated now (Dec. 2005) but I did not see it mentionned on this forum.

" }-

One thing to keep in mind is the relatively small cross section of threat types that are collected using a nepenthes sensor (because it only collects from a limited number of known Windows vulnerabilities) - ie almost exclusively backdoors (IRCbot/Rbot/SdBot) and net-worms (how much Korgo/Padobot!!). Also, many samples are collected very early in their lifespan and detections often improve dramatically after 24/48 hours.

I normally scan all new samples from my nepenthes sensor on Jotti's and often find samples only detected by 1 or 2 AVs, but after submission detection is quickly added by many vendors (within 24-48 hours).

Ned

[Tweakie]

-{ Quote: "One thing to keep in mind is the relatively small cross section of threat types that are collected using a nepenthes sensor (because it only collects from a limited number of known Windows vulnerabilities)(...)
Ned" }-

Yes, and I do not see any easy way to develop such an automatic sensor for simulating IE vulnerabilities and browsing malicious websites (source of most adwares/spywares) or to classify automatically P2P malware...

[Marcos]

-{ Quote: "
The test is available here :
http://nepenthes.mwcollect.org/stats:scannertest
" }-

It'd be interesting to know what version of NOD32 you are using in order to know whether AH, runtime packers and archives were enabled by default.

[IBK]

if i remember fine, nepenthes counts only exact detections, detections like "unknown virus" would not be counted as detected.

[StevieO]

As Ned Slider pointed out regarding the Nepenthes sensor

"(because it only collects from a limited number of known Windows vulnerabilities) - ie almost exclusively backdoors (IRCbot/Rbot/SdBot) and net-worms (how much Korgo/Padobot!!). Also, many samples are collected very early in their lifespan and detections often improve dramatically after 24/48 hours."

Well excuse me for speaking out of turn, but isn't this exactly what we want our AV's etc to do. A early detection as possible, that's what i want anyway, i don't know about everybody else !

So looking at the chart from those tests, at that moment in time anyway, who would you like to have in your PC ?

1 Antivir 99,04% +7,07%
2 BitDefender 96,23% +1,52%
3 VirusBlockAde 95,17% +1,42%

Maybe not the one you have now ! Lucky me i've got the top 2 in mine.

Yes i'm perfectly aware that it was a snapshot, but non the less i'm sure lots of people will be very surprised with the results. Look where you know who and you know who are placed !

It would be very interesting to see a brand new test done, and how they all now compare.

StevieO

[Marcos]

I too would like to see a test with AH, runtime packers and archives enabled, otherwise the test results cannot be treated seriously.

[Ned Slider]

As StevieO correctly recognizes, it's only a snapshot.

I see many samples that I collect on my nepenthes sensor that are only detected initially by Dr. Web, but yet I still choose KAV as my AV. Why, because that's just one class of sample and I don't base my decision on a single case.

And another point, all of these threats identified by nepenthes may simply be totally eliminated by being fully patched or running a firewall - you don't actually need an AV to protect against any of them. So, IMHO it's a somewhat pointless discussion.

Early detection though is something that's very difficult to quantify. For example, AV-Comparatives uses ~250,000 samples excluding Dos and other malwares, but how many do you think are recent (ie, collected within the last 24-48 hours). Virtually none relative to the sample set, so a product could easily achieve a score of 99% but fail to detect any samples identified within the last week. I think the most we can expect is that an AV vendor adds detections quickly when submitted to them. So long as AV products use definitions, they're always going to be playing catchup and with the rate new malware is appearing (approximating Moores Law - doubling every 18 months) it's only going to get harder. You have to wonder at what point the smaller companies are no longer going to be able to keep up.

Ned

[Marcos]

I'd merely wanted to add that we've just discovered that NOD32 running at Jotti's hasn't been using heuristics for quite long time. We are now working with them on a fix, so take these results with a pinch of salt.

[Gavin - DiamondCS]

I can again confirm what Happy Bytes, Marcos and others are saying. Lots of files are installers, broken files etc. An AV which detects an incomplete file could just be using something at the top of the file to detect. A tester does NOT know unless they take the extreme time to analyse the sample completely x 100,000 = long time ;D

Michael may I have that Upack file please ;)

[Firefighter]

-{ Quote: "I'd merely wanted to add that we've just discovered that NOD32 running at Jotti's hasn't been using heuristics for quite long time. We are now working with them on a fix, so take these results with a pinch of salt." }-Let's hope at least that these snapshots taken from my last test were as a result of some kind of heuristics.

Best reagrds,
Firefighter!

[Happy Bytes]

-{ Quote: "Michael may I have that Upack file please ;)" }-

As far as i remember it was "WinUpack 0.31 beta" lemme check the MD5 of it...
MD5 is: 9fd5d0445992ed686d31558f4438448c

Kaspersky fixed this f/p ( i'm not suprised, becaus i know that Roel was reading this thread ;D ) ClamAV, Fortinet, VBA32 and ArcaVir still detecting it. As for ArcaVir it's another case - it was in the first instance detected via signature - they removed it - and now detected via heuristics.

[Marcos]

-{ Quote: "Let's hope at least that these snapshots taken from my last test were as a result of some kind of heuristics.
" }-

You are refering to ThretSense and Advanced heuristics. I meant standard heuristics which can detect another big bunch of nasties without the appropriate signature.

[Sandish]

-{ Quote: "
ClamAV is producing this false positive because they created a signature over the Upack unpack stub. (Worst Chase)
QuickHeal (DNA-Scan) flags even a wet poop if it's runtime compressed
Kaspersky is the initiator of this Signature False Positive and the rest (that's another sad story) only includes it because Kaspersky detects it. If someone would have analyzed this file in a proper way they would have noticed that it's NOT malware." }-

It´s realy funny to read such comments, as if NOD32 is gods gift to the AV industry. Come on, not every signature in your database is based on a strict scientific aproach, not failsafe and sometimes it´s realy ridiculous how you and other vendors try to identifiy malware. Matching a packer is one thing, matching plain textfiles as a usualy binary based trojan is another. And i refuse to give a public example - feel free to pm me.

[Firefighter]

-{ Quote: "And now the question is how much would you trust sentences like "The samples where detected by at least 2 different scanners"." }-That's why I wrote in my report.

"...there have to be AT LEAST TWO detections to minimize those False Alarms!" so, there was not that, "...to stop those False Alarms!" ::)

I believe of course that your example was only one among those many others, which were real infected ones.

Best regards,
Firefighter!

[Marcos]

Nope, popcaploader is another suchlike example. Should we search for more? :-)

[Firefighter]

-{ Quote: "Nope, popcaploader is another suchlike example. Should we search for more? :-)" }-If you wish, but check out the % (or is it better to use ‰ ?) of False Alarms too picked from Jotti's snapshots detected by AT LEAST TWO scanners as well, so that we all can get a bit more information from Jotti's.

Best regards,
Firefighter!

[ronjor]

Off topic post removed. Let's avoid these type posts on Wilders forums.

[Marcos]

False positives may comprise a very small portion out of all samples tested, but how many of them were undetected installers and the threats were actually detected during installation? I for one found about 1000 installers in a week and this is not a small number compared to those 6x100 whose scan result screenshots you captured. Hence I'm saying the results must be taken with a lump of salt.

[Firefighter]

-{ Quote: "I for one found about 1000 installers in a week and this is not a small number compared to those 6x100 whose scan result screenshots you captured." }-Thank's about that. If that 1k is about the total amount of installers scanned in Jotti's per a week, I'd say that in my 6x100 snapshot sample collection were about 3...8 % installers. This estimate was taken from those hours spent to finish that 6x100 snapshot collection. Here you can see what kind of snapshots I mean.

Best regards,
Firefighter!

PS. Just checked from the snapshot sample names, where were some of these words, "setup", "install" and "installer", and I got 5.3 % of them from those 6x100 snapshots.

[minceypw]

-{ Quote: "I'd merely wanted to add that we've just discovered that NOD32 running at Jotti's hasn't been using heuristics for quite long time. We are now working with them on a fix, so take these results with a pinch of salt." }-

Hi Marcos

Has Jotti's now setup NOD32 correctly.

[Marcos]

Nope, many samples are not shown as detected though they actually are. From the service provider I got a reply that it's due to memory exhaustion caused by AV scanners that scan files before NOD32. Sometimes submitting an undetected file again will show it as detected.

[pykko]

-{ Quote: "Nope, many samples are not shown as detected though they actually are. From the service provider I got a reply that it's due to memory exhaustion caused by AV scanners that scan files before NOD32. Sometimes submitting an undetected file again will show it as detected." }-

And he can't fix this problem somehow?
All other AVs from NOD32 to the end of the scanners are affected I guess.. :(
I have a suggestion...one month NOD32, VBA32, Norman to be the first scanners, and one month the firt: AntiVir, Avast, BD...etc . ;D ;D

[Firefighter]

-{ Quote: "Nope, many samples are not shown as detected though they actually are. From the service provider I got a reply that it's due to memory exhaustion caused by AV scanners that scan files before NOD32. Sometimes submitting an undetected file again will show it as detected." }-In my mind that test was only a test about Jotti's. I have found several samples detected by DrWeb 4.33.2 but not with DrWeb in Jotti's albeit DrWeb was the best in my Jotti's 6 x 100 comparison.

Best regards,
Firefighter!