The New Anti-Virus Test - Packers Support

Hi everybody!

We made our own antivirus test - Packers Support Test. It was made by www.anti-malware.ru, the independent research project.

We chose 21 packers that are the most popular among viruswriters:

1. ACProtect 1.32
2. ASPack 2.12
3. ASProtect 2.1 buid 2.19
4. Dropper 2.0
5. EXECryptor 2.3.9.0
6. ExeStealth 2.76
7. FSG 2.0
8. MEW 11 SE 1.2
9. Morphine 2.7
10. NsPack 3.7
11. Obsidium 1.2.5.0
12. ORiEN 2.12
13. Packman 1.0
14. PECompact2 2.78a
15. PESpin 1.304
16. Petite 2.3
17. Private exe Protector 1.9
18. UPX 2.01w
19. WinUpack 0.39 final
20. yoda's Cryptor 1.3
21. yoda's Protector 1.0b

and used them to pack 8 malware samples (had been choosen by rendom, unpacked of course)

* Backdoor.Win32.BO_Installer
* Email-Worm.Win32.Bagle
* Email-Worm.Win32.Menger
* Email-Worm.Win32.Naked
* Email-Worm.Win32.Swen
* Worm.Win32.AimVen
* Trojan-PSW.Win32.Avisa
* Trojan-Clicker.Win32.Getfound


So this's their results:

Gold Packers Support
F-Secure Anti-Virus 2006 - 81%
Kaspersky Anti-Virus 6.0 - 81%

Silver Packers Support
BitDefender 9 Professional Plus - 76%
Dr.Web Anti-Virus 4.33 - 76%

Bronze Packers Support
Eset NOD32 Antivirus 2.5 - 57%

All other AV was failed

Full test results and methodology published here
http://www.anti-malware.ru/index.phtml?part=tests
http://www.anti-malware.ru/index.phtml?part=tests&test=packers_test1

Detailed result in PDF are also available there.
But it published in russian (english page is still under construction), so you need to use http://www.google.com/language_tools for translation.

 

Interesting test, but I find the evaluation rather questionable. I mean, would you really prefer ClamAV (with 2/21) over Norton (1/21) for example, when the number of unpacked files is 26 for Clam and 83 for Norton (I may have miscounted a few)?

 

Correct me if I'm wrong, but is this test not creating/modifying malware simply for testing? What do these results show in the "real-world"

Regards

 

That's a good question, of course.
I was just trying to say that in my opinion, the number of unpacked files here is probably more important for the real world than the number of "100%-supported" packers.

 

No. of "YES": [from the excel file =COUNTIF($A$2:$K$28,"YES")]

avast! Antivirus 79
AVG Antivirus 68
Avira Antivir 55
BitDefender Antivirus 122
CA eTrust Antivirus 24
ClamAV Antivirus 34
Dr.Web Antivirus 119
Eset NOD32 Antivirus 123
F-Secure Antivirus 124
Kaspersky Anti-Virus 124
McAfee VirusScan 61
Panda Antivirus 45
Sophos Antivirus 54
Symantec Antivirus 92
Trend Micro Antivirus 26
UNA Antivirus 38
VBA32 Antivirus 83

 

Simple question - how do you know for sure that the antivirus programs really unpacked the malware? They could have a detection that simply catches the packed variant of the malware.

And even if the packer can be unpacked, how you know for sure that antivirus program can *not* do it, they maybe just missed the sample because the unpacked file has structures so different that the normal detection doesn't fit.

There is no way to tell unless to reverse engineer the antivirus program.

ESET has 123 * YES and 57%, Bitdefender has 122 * YES and 76%... Interesting math involved here.
I wonder if the tester enabled all options (emulation, heuristics etc.) and also rated detection achieved with that.

 

Not to mention those AVs that i.e. flag everything packed by a certain runtime packer.... looking at the table Morphine is a likely candidate for many...

Btw. avira has detected all fsg files, yet gets a failed in the summary?

Edit: seems the results got updated, or i was too tired early in the morning....

 

Just looked at the XLS result table, the test is flawed as I suspected. It lists several packers as not supported, but I know that Avira does - I wrote some of the unpacker modules by myself. It also lists support for packers that Avira does not support currently.

 

This test showes what packers are supported by popular anti-virus engines.
Sometimes viruswriters just repack old virus and it will be "new" for many AV if they don't support new packer.

I think packers support is very important, especially for gateway products.

 

It don't seems suspected. We discussed this on anti-malware.ru forum, it can seldom be because of not precise signatire for the particular virus.

 

Well, gateway products just report every runtime packed file as suspicious, that's enough. Just look at Sophos, they recently added this behaviour and are happy with it.

The test is useless - it cannot proof that an antivirus engine can or cannot unpack a certain runtime packer.

Nor all protection methods were enabled or counted (heuristics etc.). The users don't care if a malware was unpacked if it got detected anyway.

 

I'm really interested in knowing whether they had Advanced heuristics in NOD32 enabled which uses a generic unpacker. Did they also test such packed files for functionality, just to be 100% sure they work?

 

Marcos, heuristics in NOD32 was enabled and all packed samples was tested for malicious functionality (see the first table in XLS of PDF file).

 

IlyaOS: Will you share the samples with AV companies? I'm sure that despite the critizism they are interested in the samples to check out why the samples where missed even though the packers are supported. Might be interesting for you as well.

I only know the address for avira for such cases, which would be heuristik2 [at] avira.com

 

FRug: Yes, we shared all packed samples with some vendors who had been interested in this test. They're using them both to find "bugs" in packers support and in writing signatures process.

Thanks for Avira email.

 

Well, AntiVir's heuristic was not enabled (or rated), as the result table obviously show.

 

All options in AntiVir was enabled, we used maximum protection level in all tested antiviruses! The Results just show that in packers support in AntiVir works incorrectly or this AV contain some inaccurate signatures (as minimum).
Anyway its indifferently for users, repacked virus can't be detected.

 

IlyaOS: Just as Stefan said, I believe some files were actually unpacked, but the result didn't "match" the way the particular detection method works. You are right that it doesn't matter for the users - but you draw conclusions about the packer support based on these data
Anyway, I would like to check those samples... is it possible to upload them to ftp://ftp.avast.com/incoming, please? (avast! FTP, no read/list rights - upload only).
Thanks!

 

Complety useless and proves that the testers do not understand how antivirus engines can work. Here are some short facts to think about:

Almost every AV Vendor is able to make so called "Mass-Adding" for Malware. There are different methods for this for example Parts of Section CRC or even full file CRC for trivial files. If you do so you will have for sure problems detecting this in a UPack file - EVEN IF YOU CAN PROPER UNPACK UPACK - due to the fact that Upack Unpacking produces unaligned section data if unpacked via Emulation. Another fact is that some packers merging several sections into less but bigger sections. So it is FOR SURE THEN that these files will not be detected if added via the mass adding what doesn't confirm that you are unable to unpack this particular packer. You have to know how the vendor added the signature in the first instance before you can make any assumption and this requires reverse engineering of the virus database or at least fundamendal understanding of AV engines what you guys don't have as it seems because otherwise you wouldn't come up with such an pumpkin idea to test av unpacking capatibilities like this.

Next thing is - Stefan explained this already - that some vendors flaging based on packer - Sophos for example will successful flag every single UPack File as MAL/Packer. That doesn't mean that they can unpack this - otherwise they wouldn't need this "dirty" trick for a desktop system to flag every upack file - EVEN THE RUNTIME PACKER ITSELF - that is for sure not malicious!

I'm getting tired of explaining day by day to wannabe antivirus testers how av testing SHOULD work. Just buy the book next year and read it.

Mike

 

I know we're getting off-topic, but I must say this sentence doesn't make much sense to me...

 

Depends on Emulation, but you might hunt for correct imports manualy after this.

Edit: PM sent that we don't go offtopic.

 

Inspector Clouseau, all "facts" that you wrote above belong to viruslabs and technologies of some AV vendors. I don't think that permanent adding signatures of packed viruses to database is good idea, much better to support more packers, dosn't it?

By the way, Kaspersky, BitDefender and DrWeb passed our test without any problems, so maybe the thing is that just some AV engines or viruslabs technologies are not up-to-date?

To flag every packed file (e.g. upack) as a malicious is really bad idea, seems like "impotence".

 

IlyaOS, how do you prove that a file that was reported as infected actually got unpacked? And how you prove that a file that did not got reported was not unpacked?

Yes, unpacking is important, but the way you tested it does not prove if an antivirus program can or cannot unpack that specific packer. The testing method is completely flawed.

And again I repeat: you did not use the maximum protection settings for Avira, otherwise lots of the files would have been reported as suspicious.

Maybe it is safe to assume that you just want to prove that .ru antivirus products are superior?

 

You don't get it what proves that you did not even understand about what i was talking. I'm not sure if i should continue going on to try to explain to you simple and basic av procedures if you do not even understand the very basic facts.

Mass adding Malware doesn't automatically mean that you have to add every file runtime packed. As i said before you can add a UNPACKED FILE via section CRC or full file CRC. Which makes sure that this file gets detected as it is what is just fine for non-important samples if it's just "another" sample which you have to detect. Once again: That's up to the Vendor what he judges as important and adds it via several methods to make sure a sucessful detection. You are not in this liga now where you can make this conclusion what is really important to detect in various patched ways if i take a look to your random picked samples. You even wrote they are random picked - this itself is the major flaw! Once again (and please read this twice before you reply) Detecting a unpacked sample and not detecting a packed sample DOES NOT PROVE IF YOU CAN UNPACK THIS PARTICULAR PACKER. For the reasons i explained in my previous post. Some of the runtime packers do extend/merge sections where you CANNOT USE section CRC's or full file CRC's since they will look completely different after unpacking. You have to use there for example a entry point depending offset to look for a particular signature since the entrypoint holds always the correct position for the startup opcode which leads you to the correct instruction flow. Is this really so difficult to understand for you? If yes please stop antivirus testing it doesn't make any sense.

 

Hmm...You just went too far with that comment. Very naive:


1. Kaspersky:


Based in Moscow, Russia.

http://www.kaspersky.com/about


2. F-Secure:


Based in Helsinki, Finland.

http://www.f-secure.com/f-secure/


3. BitDefender


Based in Bucarest, Romania.

http://www.bitdefender.com/site/view/company.html


4. Dr.Web


Based in Saint-Petersburg & Moscow, Russia.

http://company.drweb.com/about/


5. Eset-NOD32


Based in Bratislava, Slovakia.

http://www.eset.com/company/index.php


/////////////////////////////////////////////////

I only see one Romanian AV product in here and, by the way, I am only aware of one vendor of this nationality (the other one is RAV but was acquired by Microsoft back in 2003). So where are the rest??

These kinds of comments do not show the required level of professionalism in a Computer Security forum.


Regards.