The New Anti-Virus Test - Packers Support

Discussion in 'other anti-virus software' started by IlyaOS, Oct 9, 2006.

Thread Status:
Not open for further replies.
  1. IlyaOS

    IlyaOS Registered Member

    Joined:
    Nov 13, 2005
    Posts:
    29
    Hi everybody!

    We made our own antivirus test - Packers Support Test. It was made by www.anti-malware.ru, the independent research project.

    We chose 21 packers that are the most popular among viruswriters:

    1. ACProtect 1.32
    2. ASPack 2.12
    3. ASProtect 2.1 buid 2.19
    4. Dropper 2.0
    5. EXECryptor 2.3.9.0
    6. ExeStealth 2.76
    7. FSG 2.0
    8. MEW 11 SE 1.2
    9. Morphine 2.7
    10. NsPack 3.7
    11. Obsidium 1.2.5.0
    12. ORiEN 2.12
    13. Packman 1.0
    14. PECompact2 2.78a
    15. PESpin 1.304
    16. Petite 2.3
    17. Private exe Protector 1.9
    18. UPX 2.01w
    19. WinUpack 0.39 final
    20. yoda's Cryptor 1.3
    21. yoda's Protector 1.0b

    and used them to pack 8 malware samples (had been choosen by rendom, unpacked of course)

    * Backdoor.Win32.BO_Installer
    * Email-Worm.Win32.Bagle
    * Email-Worm.Win32.Menger
    * Email-Worm.Win32.Naked
    * Email-Worm.Win32.Swen
    * Worm.Win32.AimVen
    * Trojan-PSW.Win32.Avisa
    * Trojan-Clicker.Win32.Getfound


    So this's their results:

    Gold Packers Support
    F-Secure Anti-Virus 2006 - 81%
    Kaspersky Anti-Virus 6.0 - 81%

    Silver Packers Support
    BitDefender 9 Professional Plus - 76%
    Dr.Web Anti-Virus 4.33 - 76%

    Bronze Packers Support
    Eset NOD32 Antivirus 2.5 - 57%

    All other AV was failed :(

    Full test results and methodology published here
    http://www.anti-malware.ru/index.phtml?part=tests
    http://www.anti-malware.ru/index.phtml?part=tests&test=packers_test1

    Detailed result in PDF are also available there.
    But it published in russian (english page is still under construction), so you need to use http://www.google.com/language_tools for translation.
     
  2. i_g

    i_g Registered Member

    Joined:
    Aug 30, 2006
    Posts:
    133
    Interesting test, but I find the evaluation rather questionable. I mean, would you really prefer ClamAV (with 2/21) over Norton (1/21) for example, when the number of unpacked files is 26 for Clam and 83 for Norton (I may have miscounted a few)?
     
  3. Londonbeat

    Londonbeat Registered Member

    Joined:
    Sep 21, 2006
    Posts:
    350
    Correct me if I'm wrong, but is this test not creating/modifying malware simply for testing? What do these results show in the "real-world"o_O

    Regards
     
  4. i_g

    i_g Registered Member

    Joined:
    Aug 30, 2006
    Posts:
    133
    That's a good question, of course.
    I was just trying to say that in my opinion, the number of unpacked files here is probably more important for the real world than the number of "100%-supported" packers.
     
  5. apm

    apm Registered Member

    Joined:
    Mar 15, 2006
    Posts:
    162
    No. of "YES": [from the excel file =COUNTIF($A$2:$K$28,"YES")]

    avast! Antivirus 79
    AVG Antivirus 68
    Avira Antivir 55
    BitDefender Antivirus 122
    CA eTrust Antivirus 24
    ClamAV Antivirus 34
    Dr.Web Antivirus 119
    Eset NOD32 Antivirus 123
    F-Secure Antivirus 124
    Kaspersky Anti-Virus 124
    McAfee VirusScan 61
    Panda Antivirus 45
    Sophos Antivirus 54
    Symantec Antivirus 92
    Trend Micro Antivirus 26
    UNA Antivirus 38
    VBA32 Antivirus 83
     
  6. Stefan Kurtzhals

    Stefan Kurtzhals AV Expert

    Joined:
    Sep 30, 2003
    Posts:
    701
    Simple question - how do you know for sure that the antivirus programs really unpacked the malware? They could have a detection that simply catches the packed variant of the malware.

    And even if the packer can be unpacked, how you know for sure that antivirus program can *not* do it, they maybe just missed the sample because the unpacked file has structures so different that the normal detection doesn't fit.

    There is no way to tell unless to reverse engineer the antivirus program.

    ESET has 123 * YES and 57%, Bitdefender has 122 * YES and 76%... Interesting math involved here. :)
    I wonder if the tester enabled all options (emulation, heuristics etc.) and also rated detection achieved with that.
     
  7. FRug

    FRug Registered Member

    Joined:
    Feb 7, 2006
    Posts:
    309
    Not to mention those AVs that i.e. flag everything packed by a certain runtime packer.... looking at the table Morphine is a likely candidate for many...

    Btw. avira has detected all fsg files, yet gets a failed in the summary?

    Edit: seems the results got updated, or i was too tired early in the morning....
     
    Last edited: Oct 10, 2006
  8. Stefan Kurtzhals

    Stefan Kurtzhals AV Expert

    Joined:
    Sep 30, 2003
    Posts:
    701
    Just looked at the XLS result table, the test is flawed as I suspected. It lists several packers as not supported, but I know that Avira does - I wrote some of the unpacker modules by myself. :) It also lists support for packers that Avira does not support currently. :rolleyes:
     
  9. IlyaOS

    IlyaOS Registered Member

    Joined:
    Nov 13, 2005
    Posts:
    29
    This test showes what packers are supported by popular anti-virus engines.
    Sometimes viruswriters just repack old virus and it will be "new" for many AV if they don't support new packer.

    I think packers support is very important, especially for gateway products.
     
  10. IlyaOS

    IlyaOS Registered Member

    Joined:
    Nov 13, 2005
    Posts:
    29
    It don't seems suspected. We discussed this on anti-malware.ru forum, it can seldom be because of not precise signatire for the particular virus.
     
  11. Stefan Kurtzhals

    Stefan Kurtzhals AV Expert

    Joined:
    Sep 30, 2003
    Posts:
    701
    Well, gateway products just report every runtime packed file as suspicious, that's enough. Just look at Sophos, they recently added this behaviour and are happy with it.

    The test is useless - it cannot proof that an antivirus engine can or cannot unpack a certain runtime packer.

    Nor all protection methods were enabled or counted (heuristics etc.). The users don't care if a malware was unpacked if it got detected anyway.
     
  12. Marcos

    Marcos Eset Staff Account

    Joined:
    Nov 22, 2002
    Posts:
    14,374
    I'm really interested in knowing whether they had Advanced heuristics in NOD32 enabled which uses a generic unpacker. Did they also test such packed files for functionality, just to be 100% sure they work?
     
  13. IlyaOS

    IlyaOS Registered Member

    Joined:
    Nov 13, 2005
    Posts:
    29
    Marcos, heuristics in NOD32 was enabled and all packed samples was tested for malicious functionality (see the first table in XLS of PDF file).
     
  14. FRug

    FRug Registered Member

    Joined:
    Feb 7, 2006
    Posts:
    309
    IlyaOS: Will you share the samples with AV companies? I'm sure that despite the critizism they are interested in the samples to check out why the samples where missed even though the packers are supported. Might be interesting for you as well.

    I only know the address for avira for such cases, which would be heuristik2 [at] avira.com
     
  15. IlyaOS

    IlyaOS Registered Member

    Joined:
    Nov 13, 2005
    Posts:
    29
    FRug: Yes, we shared all packed samples with some vendors who had been interested in this test. They're using them both to find "bugs" in packers support and in writing signatures process.

    Thanks for Avira email.
     
  16. Stefan Kurtzhals

    Stefan Kurtzhals AV Expert

    Joined:
    Sep 30, 2003
    Posts:
    701
    Well, AntiVir's heuristic was not enabled (or rated), as the result table obviously show.
     
  17. IlyaOS

    IlyaOS Registered Member

    Joined:
    Nov 13, 2005
    Posts:
    29
    All options in AntiVir was enabled, we used maximum protection level in all tested antiviruses! The Results just show that in packers support in AntiVir works incorrectly or this AV contain some inaccurate signatures (as minimum).
    Anyway its indifferently for users, repacked virus can't be detected.
     
  18. i_g

    i_g Registered Member

    Joined:
    Aug 30, 2006
    Posts:
    133
    IlyaOS: Just as Stefan said, I believe some files were actually unpacked, but the result didn't "match" the way the particular detection method works. You are right that it doesn't matter for the users - but you draw conclusions about the packer support based on these data ;)
    Anyway, I would like to check those samples... is it possible to upload them to ftp://ftp.avast.com/incoming, please? (avast! FTP, no read/list rights - upload only).
    Thanks!
     
  19. Inspector Clouseau

    Inspector Clouseau AV Expert

    Joined:
    Apr 2, 2006
    Posts:
    1,329
    Location:
    Maidenhead, UK
    Complety useless and proves that the testers do not understand how antivirus engines can work. Here are some short facts to think about:

    Almost every AV Vendor is able to make so called "Mass-Adding" for Malware. There are different methods for this for example Parts of Section CRC or even full file CRC for trivial files. If you do so you will have for sure problems detecting this in a UPack file - EVEN IF YOU CAN PROPER UNPACK UPACK - due to the fact that Upack Unpacking produces unaligned section data if unpacked via Emulation. Another fact is that some packers merging several sections into less but bigger sections. So it is FOR SURE THEN that these files will not be detected if added via the mass adding what doesn't confirm that you are unable to unpack this particular packer. You have to know how the vendor added the signature in the first instance before you can make any assumption and this requires reverse engineering of the virus database or at least fundamendal understanding of AV engines what you guys don't have as it seems because otherwise you wouldn't come up with such an pumpkin idea to test av unpacking capatibilities like this.

    Next thing is - Stefan explained this already - that some vendors flaging based on packer - Sophos for example will successful flag every single UPack File as MAL/Packer. That doesn't mean that they can unpack this - otherwise they wouldn't need this "dirty" trick for a desktop system to flag every upack file - EVEN THE RUNTIME PACKER ITSELF - that is for sure not malicious!

    I'm getting tired of explaining day by day to wannabe antivirus testers how av testing SHOULD work. Just buy the book next year and read it.

    Mike
     
  20. i_g

    i_g Registered Member

    Joined:
    Aug 30, 2006
    Posts:
    133
    I know we're getting off-topic, but I must say this sentence doesn't make much sense to me... :doubt:
     
  21. Inspector Clouseau

    Inspector Clouseau AV Expert

    Joined:
    Apr 2, 2006
    Posts:
    1,329
    Location:
    Maidenhead, UK
    Depends on Emulation, but you might hunt for correct imports manualy after this.

    Edit: PM sent that we don't go offtopic.
     
  22. IlyaOS

    IlyaOS Registered Member

    Joined:
    Nov 13, 2005
    Posts:
    29
    Inspector Clouseau, all "facts" that you wrote above belong to viruslabs and technologies of some AV vendors. I don't think that permanent adding signatures of packed viruses to database is good idea, much better to support more packers, dosn't it? :)

    By the way, Kaspersky, BitDefender and DrWeb passed our test without any problems, so maybe the thing is that just some AV engines or viruslabs technologies are not up-to-date? :)

    To flag every packed file (e.g. upack) as a malicious is really bad idea, seems like "impotence". :'(
     
  23. Stefan Kurtzhals

    Stefan Kurtzhals AV Expert

    Joined:
    Sep 30, 2003
    Posts:
    701
    IlyaOS, how do you prove that a file that was reported as infected actually got unpacked? And how you prove that a file that did not got reported was not unpacked?

    Yes, unpacking is important, but the way you tested it does not prove if an antivirus program can or cannot unpack that specific packer. The testing method is completely flawed.

    And again I repeat: you did not use the maximum protection settings for Avira, otherwise lots of the files would have been reported as suspicious.

    Maybe it is safe to assume that you just want to prove that .ru antivirus products are superior?
     
  24. Inspector Clouseau

    Inspector Clouseau AV Expert

    Joined:
    Apr 2, 2006
    Posts:
    1,329
    Location:
    Maidenhead, UK
    You don't get it what proves that you did not even understand about what i was talking. I'm not sure if i should continue going on to try to explain to you simple and basic av procedures if you do not even understand the very basic facts.

    Mass adding Malware doesn't automatically mean that you have to add every file runtime packed. As i said before you can add a UNPACKED FILE via section CRC or full file CRC. Which makes sure that this file gets detected as it is what is just fine for non-important samples if it's just "another" sample which you have to detect. Once again: That's up to the Vendor what he judges as important and adds it via several methods to make sure a sucessful detection. You are not in this liga now where you can make this conclusion what is really important to detect in various patched ways if i take a look to your random picked samples. You even wrote they are random picked - this itself is the major flaw! Once again (and please read this twice before you reply) Detecting a unpacked sample and not detecting a packed sample DOES NOT PROVE IF YOU CAN UNPACK THIS PARTICULAR PACKER. For the reasons i explained in my previous post. Some of the runtime packers do extend/merge sections where you CANNOT USE section CRC's or full file CRC's since they will look completely different after unpacking. You have to use there for example a entry point depending offset to look for a particular signature since the entrypoint holds always the correct position for the startup opcode which leads you to the correct instruction flow. Is this really so difficult to understand for you? If yes please stop antivirus testing it doesn't make any sense.
     
  25. Seishin

    Seishin Registered Member

    Joined:
    Aug 2, 2006
    Posts:
    204
    Hmm...You just went too far with that comment. Very naive:


    1. Kaspersky:


    Based in Moscow, Russia.

    http://www.kaspersky.com/about


    2. F-Secure:


    Based in Helsinki, Finland.

    http://www.f-secure.com/f-secure/


    3. BitDefender


    Based in Bucarest, Romania.

    http://www.bitdefender.com/site/view/company.html


    4. Dr.Web


    Based in Saint-Petersburg & Moscow, Russia.

    http://company.drweb.com/about/


    5. Eset-NOD32


    Based in Bratislava, Slovakia.

    http://www.eset.com/company/index.php


    /////////////////////////////////////////////////

    I only see one Romanian AV product in here and, by the way, I am only aware of one vendor of this nationality (the other one is RAV but was acquired by Microsoft back in 2003). So where are the rest??

    These kinds of comments do not show the required level of professionalism in a Computer Security forum.


    Regards.
     
Loading...
Thread Status:
Not open for further replies.