A question: testing unpacking support of AVs

I do not intend to sound like a moron here, and if I do then please forgive me.

Sometimes we have seen tests that occasionally gauge the unpack engine of various AV products. Due to various reasons, so far every test performed on this matter has been largely unreliable. Not that it matters a lot, but considering the amount of repacked variants making it into the Web today, I suppose a strong unpack engine (or at least good packer dependent detections) would make a decent difference in the amount of protection offered by an AV in zero-day.

Now, some people at Malware-Test lab told me a little bit about why it is difficult to reliably perform a test to gauge unpack engines of AVs, but since Malware-Test itself is not highly regarded as a reliable organization, I would like to hear the points of view of the more established testers like IBK on the affair.

 

Re: A question

We tried (and finally gave up, realizing the effort needed to make something fully reliable) to make a packer test (for worms only), with some friends. Here are some of the problems we faced.

Selecting the samples:
----------------------

- The samples should be such that:
* Every AV tested must be able to detect, using signatures, the unpacked samples.
* Samples should not be damaged by the use of the packer: they should continue to run properly and, if they spread (worms), they should spread in a packed form.
* At least some of the samples used must not be found packed ITW with some of the packers used (it is necessary to recognize AVs that sign directly packed samples from those that unpack the samples first).
* You may also want to select samples that, in their unpacked form, are heuristically detected by the various AVs (which is possible only if it is possible to distinguish detections using generic signatures from those using heuristics), which can be useful to find out if the AV emulator is able to go through the packers that are not explicitely supported.

- Problems:
* If the malware has originally been spread in a packed form, it is possible that the unpacked sample is not detected/only detected by heuristics. You should not use samples that have initialy been released in a packed form.
* Many samples will just not work once packed with some packers. You *must* test every single sample by hand, which takes a lot of time if ou do not have a correct test framework. This is the main issue.
* It's often hard to know if a given sample can also be found packed ITW. The only reliable method is then to have a sufficient number of samples to have something statistically meaningful.


Selecting the packers:
----------------------

- You may not want to use packers that are used ~only by malware authors (e.g. morphine). You would then face a situation where some AVs are detecting the packer instead of the sample, which you could not consider as a false positive.
- You must remember that for most packers, there are several versions available. If an AV is able to upack aspack v. 1.x does not mean that it is also able to unpack aspack 1.y. You must be very careful on the way you present your results.
- If you also want to test combinations of several packers, you'll soon face a combinatorial explosion of the number of samples you have to test.
- You may want to use slightly modified versions of some packers to test gerenic unpacking ability...but that's another story.
- You must not use demo versions of commercial packers (some will add a pop-up when you launch the packed file).

Testing false positives:
------------------------

You should prepare a large list of repacked legit files to sort out those AVs that flag every packed file, or that are more likely to trigger heuristics detection just because a file is packed.

Generic unpacking and heuristics:
---------------------------------

For some AVs, the emulator will be able to run through some packers but this will not be used for detecting the sample by signature. This is for example the case for Norman's Sandbox. Then, you'll see that the original sample is detected by signature and that the repacked sample is detected by heuristics. This is a valuable information by itself, but it is possible that a bigger malware, which would need more emulation cycles to be unpacked, will not be detected just because the emulator gives up too early. Then, this is difficult to quantify.

Interpreting the results:
-------------------------

Perhaps the most important thing is that raw results will not expose the valuable information. They have to be interpreted first.
- Are all working samples packed with packer X detected by AV Y ? Are they all detected by signature ? If not, does it mean that there is no unpacking support for packer X or that some specific samples/signatures are not unpacking-compatible ?
- Are all legit files packed with packer X not detected by AV Y ? If not, is that because the packer itself is detected or because heuristics are too paranoid ?
etc.

Edit: by th way, I think you should edit the title of the topic to make it more specific (like: Question : testing unpacking support of AVs)

 

You coments based on the results, but you want to test one step of the whole process, so you know, you had to enumerate all the possible and kick out all other report not triger by unpack, but in fact, the process is not linear, so in theory is not possible to make such a test.

 

It'd be interesting if it's possible to do a test like this considering some AVs have generic unpackers whilst others have a whole range of unpackers hardcoded into their products.

 

Re: A question

And that is wrong too. I'll tell you why:

You can easily add a specific sample with an unpacked signature, crc or whatsoever *AND* add an additional sig for a "pre-packed" file. Some of the runtime packers will always produce the same binary if you pack the same source (origin-file) several times. That said you can pre-pack them and add a signature for a "not yet existing" malware variant.

Then next, how do you define "Unpacking"? Some technologies requiring no working executables if they just search for a few strings or crc's in the unpack dump. Others need a fixed IAT in order to give some results. So if you don't detect anything THAT DOESN'T MEAN THAT THE AV IS NOT SUPPORTING THIS PACKER. To be very sure you would have to debug the unpacking process with every AV to see what they are doing.

Next problem: Different Versions of the Runtime Packers. For example Aspack 2.11 is "more difficult" to unpack than Aspack 2.12 because 2.11 has some (at least simple) polymorphic layer. You have to detect a few calls via "simple emulation" if you're going to unpack that for instance with static unpacking code.

Next problem: Priorities !!! It doesn't make any sense to test *Random mixed* Unpacking abilities. You should have some kind of statistic what is used often for malware. Testing old packers which aren't used anymore (maybe because they produce only working executables for Win9X) doesn't make any sense.

I could go on and on and on...

 

Re: A question

You're right, I was wondering if some AV companies did add such 'preventive' signatures. I guess this only applies to most prevalent malwares/malware families ?

That's what I meant by (I admit it was unclear): "If not, does it mean that there is no unpacking support for packer X or that some specific samples/signatures are not unpacking-compatible ?" It means that you can not gauge the support of a given runtime packer by a given AV with a simple Yes/No. If you could measure that there are X% signatures that are "unpacking-compatibles" and 100-X that are not (for a given packer), then you could rate the support of that packer by the AV. However, given all other constraints, I do not see any practical method to measure X reliably.

Unfortunately, the other option (debugging the AV), is strictly forbidden in my country (excepted for interoperability reasons) and considered as counterfeiting. There has been a famous trial exactly on that topic, that opposed an AV company to an independant researcher (I was there, and I spotted some reverse engineers in the audience).