Unpacking ability of some AV's

This test was actually a few months old. Let's assume that there are actually 1500 different (runtime)packers worldwide. When we take the reliability/confidence level to 95 %, required sample size to 21 (as it was in that test), we can get the precision/accuracy level of 21.5 %. So there is quite large error margin in this test but still NOD just can't do so much better result after all.

Best regards,
Firefighter!

 

<snip>


Oh for heavens sake! I have read and re-read those answers and I still fail to see anything there that proves NOD's unpacking abilities are better than I, and others, believe them to be.

I have had NOD's realtime scanner set to scan inside archives and set to scan all files and I have watched it completely ignore a nasty that was in a multi-file Winrar archive that I had downloaded from a friends machine. Yes it caught it as soon as I tried to extract the archive but that's not the point.

edited to remove quote of removed post - Detox

 

Well as this is "Other anti-virus software" forum I would expect a little bit of impartiallity.

edited to remove quote of removed post - Detox

 

NOD32 fanboy? I'm more with Kaspersky and i also use NOD32 along with avast! (not that much though). And NOD32 certanly isn't as weak as shown on this test. I'm perfectly sure NOD32 can unpack at least ZIP SFX (i can confirm) and UPX (have seen loads of UPX-ed files detected by NOD32). Thats 2 detections minimum. Something isn't right here. ESET guys could explain more about packers like MEW, Morphine and FSG which should most certanly be supported by NOD32 (i have seen all 3 in real world situations,samples were detected as threat).

 

Let's refrain from all this personal stuff - I'll clean out the OT personal posts and let's please stick to topic from here on out.

 

I apologize that I used the word NOD fanboy, but if those trolls were removed, I'm more pleased too.

Best regards,
Firefighter!

 

i am with rejzor on this one.
the code emulation of current nod32 should catch most packed samples, same for AH
note:
# Eset NOD32 Antivirus System 2.12.3
actually drwebs rate with current version should be higher too

i think Skeeve is right on it.

and there are 3-4 packers in the list that regularly produce corrupted exes

 

I agree with you. NOD32 was in June probably not so good as it is now, but as you say there are some things that were supported already also in June, so it looks wrong.... Maybe the tester agrees to send the used packed files to another tester so he can verify the results?

 

Even I said that this test was a few months old in my post 27. So I agree your post and the new DrWeb 4.33 engine is much better than the former 4.32b one.

Btw, SAVO rules!

Best regards,
Firefighter!

 

This is a more accurate quote of what you said.

 

Yes, 5 % + 22 % was still only 27 %, that was I meant before.

Best regards,
Firefighter!

 

Unfortunately, the statisitical analysis is by no means expected to hold. The basic assumptions are that you are drawing a random sampling from a homogeneous normally distributed population. That's by no means assured in this case. The result looks quantitative, and it is if the assumptions are strictly met, but that is a big if.

Blue

 

If someone picks up RANDOMLY 21 packers, what else this is than a normal sample testing?

Or maybe I missunderstood what you meant. Of course those packers are more or less commonly used, but this was not the purpose in this test if I understood right. In my mind this test showed only how many packers those av:s were capable to unpack overall.

Best regards,
Firefighter!

 

I'd be happy to receive these samples and test them against a June version of NOD32. I'm 100% sure it will detect much more than 5% out of the samples used with AH enabled.

The results are deceiving, misleading and unfair if the tester had runtime packers and AH disabled.

 

If you go into a parking lot, and randomly select 21 cars, do you have a random sample of the worldwide distribution of automobiles? No. There are additional local factor which bias the population (country you reside in, local income levels, access to alternate forms of transportation, need I go on....). This sampling is no different. Too many local biases are operational to apply the methods you apply.

The analysis is inappropriate to the case being discussed. For all I know, the tester sampled unpackers "in the wings" of the distribution or ran a biased test by using inappropriate measurement methods or test settings for the AV. Do I know that? Of course not. However, my own perception is that the result is suspect based on simple rank ordering performance metrics accessible to anyone using google.com

Blue

 

I have a question that I hope won't be construed as OT. Namely, do the av-comparative tests include any runtime-packed viruses?

I ask the preceding question because av-comparative's tests show excellent results for NOD and, for that reason, NOD is on my short list to replace the AV I now use IF replacement should ever become warranted.

 

To be quite honest, NOD32 does detect malware infected files that are packed with UPX, AsPack, YodaCrypt, FSG, Petite, EXEStealth, PKLite and some others to say the least. I personally tested UPX, ASPack and YodaCrypt, and it does detect malware packed with these packers.

NOD32's unpack engine is not at all bad. This test is old, and maybe NOD32 2.12 may not have had a good unpack engine (I only started using NOD32 from 2.5), but NOD32 2.5 is quite good.

As far as KAV goes, no comments - Its always had the best unpack engine
BitDefender (IMO) is second best when it comes to unpack engine.

 

I think Marcos is thinking in the right direction. NOD32 has option to disable runtime packers scanning and AH. If you disable these two (especially the first one) NOD32 won't detect anything except non packed malware. And since only NOD32 feature so detailed engine tunning my best bet is that they simply disabled this or forgot to enable it properly...

 

Yes, of course.

 

Well, the main question remains unanswered. How can you actually test the unpacking ability of some av product? Simply packing some unpacked malware with different runtime packers is *not* a valid test. As I said before, for ITW malware, some AV companies might have created those samples internally on their own, and added signatures for the repacked files aswell as the original file.

So how do you know if the malware was detected after unpacking or is actually detected in it's packed state?

 

Apart from any tooing and froing around "which is best", Blue has made a very lucid and concise deconstruction of the statistical shortcomings of the sample analysis presented above.

Very nice!

Regards

 

Well question for this also rises on what happens when packing order is changed?

Lets say sample 1 is first packed with UPX and then with Morphine.
Sample 2 is first packed with Morphine and then with UPX.

Now if we create static signatures for sample 1 (UPX and then Morphine), second (sample 1) shouldn't be detected (Morphine and then UPX) since file "structure" would look completely different. Unless AV vendors have time and resources to create massive numbers of packing possibilities (very unlikely imo).

You can replace UPX and Morphine with any other packer or crypter. Above two were meant just as an example. Or are there any limitations in multipacking samples and order of packers/crypters used.

I'm open for any corrections regarding this "theory" from AV experts (like Stefan,Marcos or IBK...)

 

I don't think it's possible to substitute unpacking capabilities by adding artificially created packed signatures. Many packers (e.g. Morphine, Yoda, etc.) use some kind of random encryption - so if you pack a file twice with the same packer, you'll get different results. It's not possible to generate the signatures in advance for such packers.
And even for the "stable packers" (that generate always the same result for the same input file), it's often enough to change one insignificant byte at the beginning of the file, and the packed result is completely different again.

 

Thanks for an advice. I only thought that the web was boundless, but if that has national/local borders, after that I'm speechless.

I have to admit that some av:s are difficult to configure right.

Best regards,
Firefighter!

 

All of this analysis and what's been forgotten is the fact that each AV analyzed the SAME FILES.

Some AV's did better than others, some did miserably.