2006-03 AV-test from Jotti's 6 x 100 snapshots

Discussion in 'other anti-virus software' started by Firefighter, Mar 17, 2006.

Thread Status:
Not open for further replies.
  1. Firefighter

    Firefighter Registered Member

    Joined:
    Oct 28, 2002
    Posts:
    1,670
    Location:
    Finland
    Hi again. Because so many av:s had upgraded their engines since last fall, I made a new Jotti's av-test against snapshots. The brand new Jotti's av-test with the new specifications, there have to be AT LEAST TWO detections to minimize those False Alarms!

    2006-03 AV-test from Jotti's 6 x 100 snapshots:

    All samples in these snapshots are NOT with "ZIP", "RAR", "CAB" format or those "COM" format, that are actually old DOS samples. (most of all are with "EXE" format).

    Those snapshots were excluded, where all av:s were capable to detect the sample, but there have still to be at least two detectings in each snapshot.

    Checked as viruses/worms:

    Total ------ Set 1 ------ Set 2 ----- Set 3 ----- Set 4 ---- Set 5 ----- Set 6

    _6.3 % ----- _6 % ----- _4 % ----- 12 % ----- _3 % ----- _8 % ----- _5 %

    ================================================================================

    Detection rate:

    Total ------ Set 1 ------ Set 2 ----- Set 3 ----- Set 4 ---- Set 5 ----- Set 6


    72.3 % ----- 82 % ----- 69 % ----- 78 % ----- 72 % ----- 61 % ----- 72 % -- DrWeb 4.33
    72.2 % ----- 78 % ----- 76 % ----- 64 % ----- 71 % ----- 74 % ----- 70 % -- Kaspersky
    70.3 % ----- 70 % ----- 71 % ----- 66 % ----- 74 % ----- 68 % ----- 73 % -- Vba32
    62.3 % ----- 62 % ----- 57 % ----- 59 % ----- 74 % ----- 63 % ----- 59 % -- AntiVir
    59.0 % ----- 71 % ----- 63 % ----- 52 % ----- 54 % ----- 60 % ----- 54 % -- NOD32
    58.3 % ----- 65 % ----- 63 % ----- 55 % ----- 58 % ----- 55 % ----- 54 % -- BitDefender
    46.0 % ----- 55 % ----- 43 % ----- 38 % ----- 49 % ----- 49 % ----- 42 % -- Fortinet
    45.3 % ----- 51 % ----- 47 % ----- 43 % ----- 43 % ----- 52 % ----- 36 % -- Norman VC
    43.2 % ----- 49 % ----- 46 % ----- 33 % ----- 55 % ----- 41 % ----- 35 % -- ArcaVir
    42.5 % ----- 53 % ----- 50 % ----- 39 % ----- 42 % ----- 41 % ----- 30 % -- AVG
    32.8 % ----- 40 % ----- 35 % ----- 36 % ----- 35 % ----- 25 % ----- 26 % -- Avast
    28.0 % ----- 26 % ----- 28 % ----- 32 % ----- 34 % ----- 30 % ----- 18 % -- ClamAV
    24.3 % ----- 37 % ----- 27 % ----- 23 % ----- 24 % ----- 23 % ----- 12 % -- F-Prot
    24.0 % ----- 35 % ----- 20 % ----- 20 % ----- 31 % ----- 17 % ----- 21 % -- UNA
    17.0 % ----- 26 % ----- 19 % ----- 21 % ----- 16 % ----- 12 % ----- 08 % -- VirusBuster

    ================================================================================

    Here are those ProActive like detections:

    ProActive (heuristics + behaves like + based + BACKDOOR.Trojan + DLOADER.Trojan+ DLOADER.IRC.Trojan + GenPack: + MULDROP.Trojan + STPAGE.Trojan + Win32:Trojan-gen + WIN.IRC.WORM.Virus + Crack + fam/family + gen/generic + modified + probably + variant etc.) detection:

    Total ------ Set 1 ------ Set 2 ----- Set 3 ----- Set 4 ---- Set 5 ----- Set 6

    31.7 % ----- 27 % ----- 38 % ----- 32 % ----- 31 % ----- 36 % ----- 26 % -- NOD32
    20.5 % ----- 27 % ----- 27 % ----- 14 % ----- 22 % ----- 20 % ----- 13 % -- AVG
    15.3 % ----- 10 % ----- 23 % ----- 19 % ----- _8 % ----- 13 % ----- 19 % -- BitDefender
    14.0 % ----- 18 % ----- 21 % ----- _8 % ----- 15 % ----- 10 % ----- 12 % -- Avast
    13.3 % ----- 17 % ----- 14 % ----- 18 % ----- _5 % ----- 12 % ----- 14 % -- DrWeb 4.33
    _9.7 % ----- _0 % ----- _7 % ----- 12 % ----- 14 % ----- 14 % ----- 11 % -- AntiVir
    _9.0 % ----- _6 % ----- 10 % ----- 10 % ----- _9 % ----- 12 % ----- _7 % -- Norman VC
    _7.0 % ----- _8 % ----- 12 % ----- 11 % ----- _3 % ----- _2 % ----- _6 % -- Vba32
    _6.0 % ----- _7 % ----- _8 % ----- _3 % ----- _9 % ----- _5 % ----- _4 % -- F-Prot
    _4.8 % ----- _5 % ----- _7 % ----- _6 % ----- _5 % ----- _4 % ----- _2 % -- ArcaVir
    _3.5 % ----- _2 % ----- _3 % ----- _4 % ----- _3 % ----- _4 % ----- _5 % -- Fortinet
    _3.2 % ----- _4 % ----- _6 % ----- _5 % ----- _2 % ----- _2 % ----- _0 % -- VirusBuster
    _2.8 % ----- _6 % ----- _2 % ----- _3 % ----- _3 % ----- _3 % ----- _0 % -- Kaspersky
    _2.2 % ----- _1 % ----- _3 % ----- _4 % ----- _1 % ----- _2 % ----- _2 % -- ClamAV
    _0.2 % ----- _1 % ----- _0 % ----- _0 % ----- _0 % ----- _0 % ----- _0 % -- UNA

    ================================================================================

    Here are those ProActive like detections from those files MISSED BY SIGNATURE:

    43.6 % -- NOD32
    32.5 % -- DrWeb 4.33
    26.9 % -- BitDefender
    26.3 % -- AVG
    20.4 % -- AntiVir
    19.1 % -- Vba32
    17.2 % -- Avast
    14.1 % -- Norman VC
    _9.3 % -- Kaspersky
    _7.8 % -- ArcaVir
    _7.3 % -- F-Prot
    _6.1 % -- Fortinet
    _3.7 % -- VirusBuster
    _2.9 % -- ClamAV
    _0.2 % -- UNA

    (At least ArcaVir and maybe UNA, were not capable to use their (best) heuristics in Jotti's)


    Best regards,
    Firefighter!
     
  2. Marcos

    Marcos Eset Staff Account

    Joined:
    Nov 22, 2002
    Posts:
    14,374
    Take it always with a grain of salt, a lot of files uploaded to Jotti are either installers or corrupted files. If these were filtered out, NOD32 would have much better detection rate :) Re. installers, I found about 1000 pieces in less than a week that would be detected after being unpacked.
     
  3. Happy Bytes

    Happy Bytes Guest

    I just give one example here - and just for the note - everyone who flags this file is producing a false positive because this file is innocent, i just checked it after Marcos did send it to me:

    > "winupacke.exe" file.

    > Antivirus Version Update Result
    > AntiVir 6.34.0.53 03.18.2006 Worm/Mytob.BT
    > Avast 4.6.695.0 03.17.2006 no virus found
    > AVG 718 03.17.2006 no virus found
    > Avira 6.34.0.53 03.18.2006 Worm/Mytob.BT
    > BitDefender 7.2 03.18.2006 no virus found
    > CAT-QuickHeal 8.00 03.18.2006 (Suspicious) - DNAScan
    > ClamAV devel-20060126 03.17.2006 Worm.Mytob.Gen-6
    > DrWeb 4.33 03.18.2006 no virus found
    > eTrust-InoculateIT 23.71.105 03.18.2006 no virus found
    > eTrust-Vet 12.4.2123 03.17.2006 no virus found
    > Ewido 3.5 03.18.2006 Worm.Mytob.bt
    > Fortinet 2.71.0.0 03.18.2006 W32/MyTob.BT!net
    > F-Prot 3.16c 03.17.2006 no virus found
    > Ikarus 0.2.59.0 03.17.2006 no virus found
    > Kaspersky 4.0.2.24 03.18.2006 Net-Worm.Win32.Mytob.bt
    > McAfee 4721 03.17.2006 no virus found
    > NOD32v2 1.1449 03.17.2006 no virus found
    > Norman 5.70.10 03.17.2006 no virus found
    > Panda 9.0.0.4 03.17.2006 no virus found
    > Sophos 4.03.0 03.17.2006 no virus found
    > Symantec 8.0 03.18.2006 no virus found
    > TheHacker 5.9.5.115 03.17.2006 W32/Mytob.bt
    > UNA 1.83 03.16.2006 no virus found
    > VBA32 3.10.5 03.17.2006 Net-Worm.Win32.Mytob.bt


    And now the question is how much would you trust sentences like "The samples where detected by at least 2 different scanners".

    ClamAV is producing this false positive because they created a signature over the Upack unpack stub. (Worst Chase)
    QuickHeal (DNA-Scan) flags even a wet poop if it's runtime compressed
    Kaspersky is the initiator of this Signature False Positive and the rest (that's another sad story) only includes it because Kaspersky detects it. If someone would have analyzed this file in a proper way they would have noticed that it's NOT malware.
     
  4. Stefan Kurtzhals

    Stefan Kurtzhals AV Expert

    Joined:
    Sep 30, 2003
    Posts:
    701
    This is the original UPACK compression tool, not malware.
     
  5. Happy Bytes

    Happy Bytes Guest

    Thanks for confirming this Stefan ;)
     
  6. JimIT

    JimIT Registered Member

    Joined:
    Jan 22, 2003
    Posts:
    1,035
    Location:
    Denton, Texas

    SHHH! What are you trying to do?? Impart reason to this discussion?? We need to make sure our antivirus is DOING SOMETHING!! :D :rolleyes: ;)

    It MUST be a virus, because XYZ Antivirus SAID SO!! :rolleyes:
     
  7. hemkop

    hemkop Registered Member

    Joined:
    Feb 7, 2006
    Posts:
    61
    Thanks for confirming this JimIT :p
    It is a virus right :p ghaghaghagha
     
  8. Happy Bytes

    Happy Bytes Guest

    Sm0kie, can you play somewhere else the troll please? :rolleyes:
     
  9. Firefighter

    Firefighter Registered Member

    Joined:
    Oct 28, 2002
    Posts:
    1,670
    Location:
    Finland
    I made this limitation just because certain scanners were probably detecting too many FP:s.

    Best regards,
    Firefighter!
     
  10. pykko

    pykko Registered Member

    Joined:
    Apr 27, 2005
    Posts:
    2,236
    Location:
    Romania...and walking to heaven
    Well, like Marcos said many files are not detected because not all AVs support every type of malware, and there are many FPs. I know from my experience...files detected by KAV, VBA32, Bit Defender which were only corrupted files, or incomplete virus bodies. BD even detected the virus code copied in a HTML file as infected file. :rolleyes:
     
  11. Firefighter

    Firefighter Registered Member

    Joined:
    Oct 28, 2002
    Posts:
    1,670
    Location:
    Finland
    After all, I think that ALL scanners are making FP:s sometimes, that's why at least two detectings too.

    Best regards,
    Firefighter!
     
  12. Firefighter

    Firefighter Registered Member

    Joined:
    Oct 28, 2002
    Posts:
    1,670
    Location:
    Finland
    Is SmOkie really a Swedish? Jag tror at de är bättre att jag kan sända privat besked till honom/ henne på svenska! :D

    Best regards,
    Firefighter!
     
  13. Firefighter

    Firefighter Registered Member

    Joined:
    Oct 28, 2002
    Posts:
    1,670
    Location:
    Finland
    Are you trying to say that NOD's heuristics is more accurate that the signature detectings of any other av:s? Why this, NOD was the only that was awesome in heuristics in this test.

    Best regards,
    Firefighter!
     
  14. Devil's Advocate

    Devil's Advocate Registered Member

    Joined:
    Feb 5, 2006
    Posts:
    549
    Damn newbies. Everyone knows you need detection by at least two antiviruses, 'XYZ and ABC' to rule out false positives. :)
     
  15. pykko

    pykko Registered Member

    Joined:
    Apr 27, 2005
    Posts:
    2,236
    Location:
    Romania...and walking to heaven
    Well, here's 7 AVs detecting a corrupted file as virus.
    Even if it's infected the file is damaged and can't be excuted. ;) So you can't say 4, 5, 9, AVs detected this and it'a certainly a virus. :D
     

    Attached Files:

  16. Stefan Kurtzhals

    Stefan Kurtzhals AV Expert

    Joined:
    Sep 30, 2003
    Posts:
    701
    Why, it was a malware sample, only damaged. Those cut-off corruptions often happen during mail or ftp transfer (exploit). And it is actually ok to filter such files at the gateway.

    So detection of it is ok. Those AV programs who fail to detect it rely on a signature after unpacking the sample or the part of the file which contains the signature is cut off. Unpacking doesn't work anymore because the file is corrupted.
     
  17. Tweakie

    Tweakie Registered Member

    Joined:
    Feb 28, 2004
    Posts:
    90
    Location:
    E.U.
    Another "real world" interesting test is the one performed by the nepenthes development team. It's a bit outdated now (Dec. 2005) but I did not see it mentionned on this forum.

    The intersting thing is that this test has been performed on malware captured in the wild by nepenthes sensors (and not from collector's websites such as vxh**vens) . The test has been performed on a Linux platform and it looks like not all available options (heuristics, etc) have been activated for every scanner (they do provide the command line used). Keep this in mind when looking at the results.

    As a snapshot of current malware activity, it may better reflect the real world performance of an antivirus gateway than tests that include either a huge collection of "old" malware or only new/unknown malware.

    The test is available here :
    http://nepenthes.mwcollect.org/stats:scannertest

    The summary table :

    1 Antivir 99,04% +7,07%
    2 BitDefender 96,23% +1,52%
    3 VirusBlockAde 95,17% +1,42%
    4 F-Prot 94,02% +2,39%
    4 Authentium 94,02% new
    5 Norman Virus Control 93,78% +1,19%
    6 Fortinet 87,29% +2,35%
    7 F-Secure Antivirus 85,22% +5,99%
    8 Kaspersky 85,10% +5,73%
    9 VirusBuster 82,53% +11,76%
    10 Trend Micro 76,19% +5,14%
    11 ClamAV 71,41% -0,85%
    12 NOD32 70,06% +4,05%
    13 Sophos SWEEP 68,58% +2,45%
    14 eTrust 63,97% new
     
  18. Ned Slider

    Ned Slider Registered Member

    Joined:
    Mar 24, 2005
    Posts:
    169
    One thing to keep in mind is the relatively small cross section of threat types that are collected using a nepenthes sensor (because it only collects from a limited number of known Windows vulnerabilities) - ie almost exclusively backdoors (IRCbot/Rbot/SdBot) and net-worms (how much Korgo/Padobot!!). Also, many samples are collected very early in their lifespan and detections often improve dramatically after 24/48 hours.

    I normally scan all new samples from my nepenthes sensor on Jotti's and often find samples only detected by 1 or 2 AVs, but after submission detection is quickly added by many vendors (within 24-48 hours).

    Ned
     
  19. Tweakie

    Tweakie Registered Member

    Joined:
    Feb 28, 2004
    Posts:
    90
    Location:
    E.U.
    Yes, and I do not see any easy way to develop such an automatic sensor for simulating IE vulnerabilities and browsing malicious websites (source of most adwares/spywares) or to classify automatically P2P malware...
     
  20. Marcos

    Marcos Eset Staff Account

    Joined:
    Nov 22, 2002
    Posts:
    14,374
    It'd be interesting to know what version of NOD32 you are using in order to know whether AH, runtime packers and archives were enabled by default.
     
  21. IBK

    IBK AV Expert

    Joined:
    Dec 22, 2003
    Posts:
    1,819
    Location:
    Innsbruck (Austria)
    if i remember fine, nepenthes counts only exact detections, detections like "unknown virus" would not be counted as detected.
     
  22. StevieO

    StevieO Registered Member

    Joined:
    Feb 2, 2006
    Posts:
    1,067
    As Ned Slider pointed out regarding the Nepenthes sensor

    "(because it only collects from a limited number of known Windows vulnerabilities) - ie almost exclusively backdoors (IRCbot/Rbot/SdBot) and net-worms (how much Korgo/Padobot!!). Also, many samples are collected very early in their lifespan and detections often improve dramatically after 24/48 hours."

    Well excuse me for speaking out of turn, but isn't this exactly what we want our AV's etc to do. A early detection as possible, that's what i want anyway, i don't know about everybody else !

    So looking at the chart from those tests, at that moment in time anyway, who would you like to have in your PC ?

    1 Antivir 99,04% +7,07%
    2 BitDefender 96,23% +1,52%
    3 VirusBlockAde 95,17% +1,42%

    Maybe not the one you have now ! Lucky me i've got the top 2 in mine.

    Yes i'm perfectly aware that it was a snapshot, but non the less i'm sure lots of people will be very surprised with the results. Look where you know who and you know who are placed !

    It would be very interesting to see a brand new test done, and how they all now compare.

    StevieO
     
  23. Marcos

    Marcos Eset Staff Account

    Joined:
    Nov 22, 2002
    Posts:
    14,374
    I too would like to see a test with AH, runtime packers and archives enabled, otherwise the test results cannot be treated seriously.
     
  24. Ned Slider

    Ned Slider Registered Member

    Joined:
    Mar 24, 2005
    Posts:
    169
    As StevieO correctly recognizes, it's only a snapshot.

    I see many samples that I collect on my nepenthes sensor that are only detected initially by Dr. Web, but yet I still choose KAV as my AV. Why, because that's just one class of sample and I don't base my decision on a single case.

    And another point, all of these threats identified by nepenthes may simply be totally eliminated by being fully patched or running a firewall - you don't actually need an AV to protect against any of them. So, IMHO it's a somewhat pointless discussion.

    Early detection though is something that's very difficult to quantify. For example, AV-Comparatives uses ~250,000 samples excluding Dos and other malwares, but how many do you think are recent (ie, collected within the last 24-48 hours). Virtually none relative to the sample set, so a product could easily achieve a score of 99% but fail to detect any samples identified within the last week. I think the most we can expect is that an AV vendor adds detections quickly when submitted to them. So long as AV products use definitions, they're always going to be playing catchup and with the rate new malware is appearing (approximating Moores Law - doubling every 18 months) it's only going to get harder. You have to wonder at what point the smaller companies are no longer going to be able to keep up.

    Ned
     
  25. Marcos

    Marcos Eset Staff Account

    Joined:
    Nov 22, 2002
    Posts:
    14,374
    I'd merely wanted to add that we've just discovered that NOD32 running at Jotti's hasn't been using heuristics for quite long time. We are now working with them on a fix, so take these results with a pinch of salt.
     
Loading...
Thread Status:
Not open for further replies.