Unpacking ability of some AV's

Discussion in 'other anti-virus software' started by Blackcat, Nov 12, 2005.

Thread Status:
Not open for further replies.
  1. Firefighter

    Firefighter Registered Member

    Joined:
    Oct 28, 2002
    Posts:
    1,670
    Location:
    Finland
    This test was actually a few months old. Let's assume that there are actually 1500 different (runtime)packers worldwide. When we take the reliability/confidence level to 95 %, required sample size to 21 (as it was in that test), we can get the precision/accuracy level of 21.5 %. So there is quite large error margin in this test but still NOD just can't do so much better result after all. :rolleyes:

    Best regards,
    Firefighter!
     

    Attached Files:

  2. Patrician

    Patrician Registered Member

    Joined:
    Jun 3, 2005
    Posts:
    132
    <snip>


    Oh for heavens sake! I have read and re-read those answers and I still fail to see anything there that proves NOD's unpacking abilities are better than I, and others, believe them to be.

    I have had NOD's realtime scanner set to scan inside archives and set to scan all files and I have watched it completely ignore a nasty that was in a multi-file Winrar archive that I had downloaded from a friends machine. Yes it caught it as soon as I tried to extract the archive but that's not the point.

    edited to remove quote of removed post - Detox
     
    Last edited by a moderator: Nov 12, 2005
  3. Patrician

    Patrician Registered Member

    Joined:
    Jun 3, 2005
    Posts:
    132
    Well as this is "Other anti-virus software" forum I would expect a little bit of impartiallity.

    edited to remove quote of removed post - Detox
     
    Last edited by a moderator: Nov 12, 2005
  4. RejZoR

    RejZoR Registered Member

    Joined:
    May 31, 2004
    Posts:
    6,426
    NOD32 fanboy? I'm more with Kaspersky and i also use NOD32 along with avast! (not that much though). And NOD32 certanly isn't as weak as shown on this test. I'm perfectly sure NOD32 can unpack at least ZIP SFX (i can confirm) and UPX (have seen loads of UPX-ed files detected by NOD32). Thats 2 detections minimum. Something isn't right here. ESET guys could explain more about packers like MEW, Morphine and FSG which should most certanly be supported by NOD32 (i have seen all 3 in real world situations,samples were detected as threat).
     
  5. Detox

    Detox Retired Moderator

    Joined:
    Feb 9, 2002
    Posts:
    8,507
    Location:
    Texas, USA
    Let's refrain from all this personal stuff - I'll clean out the OT personal posts and let's please stick to topic from here on out.
     
  6. Firefighter

    Firefighter Registered Member

    Joined:
    Oct 28, 2002
    Posts:
    1,670
    Location:
    Finland
    I apologize that I used the word NOD fanboy, but if those trolls were removed, I'm more pleased too. :-*

    Best regards,
    Firefighter!
     
  7. illukka

    illukka Spyware Fighter

    Joined:
    Jun 23, 2003
    Posts:
    633
    Location:
    S.A.V.O
    i am with rejzor on this one.
    the code emulation of current nod32 should catch most packed samples, same for AH
    note:
    # Eset NOD32 Antivirus System 2.12.3

    actually drwebs rate with current version should be higher too

    i think Skeeve is right on it.

    and there are 3-4 packers in the list that regularly produce corrupted exes
     
  8. IBK

    IBK AV Expert

    Joined:
    Dec 22, 2003
    Posts:
    1,856
    Location:
    Innsbruck (Austria)
    I agree with you. NOD32 was in June probably not so good as it is now, but as you say there are some things that were supported already also in June, so it looks wrong.... Maybe the tester agrees to send the used packed files to another tester so he can verify the results?
     
  9. Firefighter

    Firefighter Registered Member

    Joined:
    Oct 28, 2002
    Posts:
    1,670
    Location:
    Finland
    Even I said that this test was a few months old in my post 27. So I agree your post and the new DrWeb 4.33 engine is much better than the former 4.32b one. :cool:

    Btw, SAVO rules!

    Best regards,
    Firefighter!
     
  10. The Hammer

    The Hammer Registered Member

    Joined:
    May 12, 2005
    Posts:
    5,716
    Location:
    Toronto Canada
    This is a more accurate quote of what you said.;)
     
  11. Firefighter

    Firefighter Registered Member

    Joined:
    Oct 28, 2002
    Posts:
    1,670
    Location:
    Finland
    Yes, 5 % + 22 % was still only 27 %, that was I meant before.

    Best regards,
    Firefighter!
     
  12. BlueZannetti

    BlueZannetti Registered Member

    Joined:
    Oct 19, 2003
    Posts:
    6,590
    Unfortunately, the statisitical analysis is by no means expected to hold. The basic assumptions are that you are drawing a random sampling from a homogeneous normally distributed population. That's by no means assured in this case. The result looks quantitative, and it is if the assumptions are strictly met, but that is a big if.

    Blue
     
  13. Firefighter

    Firefighter Registered Member

    Joined:
    Oct 28, 2002
    Posts:
    1,670
    Location:
    Finland
    If someone picks up RANDOMLY 21 packers, what else this is than a normal sample testing? :cool:

    Or maybe I missunderstood what you meant. Of course those packers are more or less commonly used, but this was not the purpose in this test if I understood right. In my mind this test showed only how many packers those av:s were capable to unpack overall.

    Best regards,
    Firefighter!
     
    Last edited: Nov 12, 2005
  14. Marcos

    Marcos Eset Staff Account

    Joined:
    Nov 22, 2002
    Posts:
    14,432
    I'd be happy to receive these samples and test them against a June version of NOD32. I'm 100% sure it will detect much more than 5% out of the samples used with AH enabled.

    The results are deceiving, misleading and unfair if the tester had runtime packers and AH disabled.
     
  15. BlueZannetti

    BlueZannetti Registered Member

    Joined:
    Oct 19, 2003
    Posts:
    6,590
    If you go into a parking lot, and randomly select 21 cars, do you have a random sample of the worldwide distribution of automobiles? No. There are additional local factor which bias the population (country you reside in, local income levels, access to alternate forms of transportation, need I go on....). This sampling is no different. Too many local biases are operational to apply the methods you apply.

    The analysis is inappropriate to the case being discussed. For all I know, the tester sampled unpackers "in the wings" of the distribution or ran a biased test by using inappropriate measurement methods or test settings for the AV. Do I know that? Of course not. However, my own perception is that the result is suspect based on simple rank ordering performance metrics accessible to anyone using google.com

    Blue
     
  16. bellgamin

    bellgamin Very Frequent Poster

    Joined:
    Aug 1, 2002
    Posts:
    6,776
    Location:
    Hawaii
    I have a question that I hope won't be construed as OT. Namely, do the av-comparative tests include any runtime-packed viruses?

    I ask the preceding question because av-comparative's tests show excellent results for NOD and, for that reason, NOD is on my short list to replace the AV I now use IF replacement should ever become warranted.
     
  17. Firecat

    Firecat Registered Member

    Joined:
    Jan 2, 2005
    Posts:
    8,175
    Location:
    The land of no identity :D
    To be quite honest, NOD32 does detect malware infected files that are packed with UPX, AsPack, YodaCrypt, FSG, Petite, EXEStealth, PKLite and some others to say the least. I personally tested UPX, ASPack and YodaCrypt, and it does detect malware packed with these packers.

    NOD32's unpack engine is not at all bad. This test is old, and maybe NOD32 2.12 may not have had a good unpack engine (I only started using NOD32 from 2.5), but NOD32 2.5 is quite good. :)

    As far as KAV goes, no comments - Its always had the best unpack engine ;)
    BitDefender (IMO) is second best when it comes to unpack engine. :)
     
  18. RejZoR

    RejZoR Registered Member

    Joined:
    May 31, 2004
    Posts:
    6,426
    I think Marcos is thinking in the right direction. NOD32 has option to disable runtime packers scanning and AH. If you disable these two (especially the first one) NOD32 won't detect anything except non packed malware. And since only NOD32 feature so detailed engine tunning my best bet is that they simply disabled this or forgot to enable it properly...
     
  19. IBK

    IBK AV Expert

    Joined:
    Dec 22, 2003
    Posts:
    1,856
    Location:
    Innsbruck (Austria)
    Yes, of course.
     
  20. Stefan Kurtzhals

    Stefan Kurtzhals AV Expert

    Joined:
    Sep 30, 2003
    Posts:
    702
    Well, the main question remains unanswered. How can you actually test the unpacking ability of some av product? Simply packing some unpacked malware with different runtime packers is *not* a valid test. As I said before, for ITW malware, some AV companies might have created those samples internally on their own, and added signatures for the repacked files aswell as the original file.

    So how do you know if the malware was detected after unpacking or is actually detected in it's packed state?
     
  21. ==001100==

    ==001100== Guest

    Apart from any tooing and froing around "which is best", Blue has made a very lucid and concise deconstruction of the statistical shortcomings of the sample analysis presented above.

    Very nice!

    Regards
     
  22. RejZoR

    RejZoR Registered Member

    Joined:
    May 31, 2004
    Posts:
    6,426
    Well question for this also rises on what happens when packing order is changed?

    Lets say sample 1 is first packed with UPX and then with Morphine.
    Sample 2 is first packed with Morphine and then with UPX.

    Now if we create static signatures for sample 1 (UPX and then Morphine), second (sample 1) shouldn't be detected (Morphine and then UPX) since file "structure" would look completely different. Unless AV vendors have time and resources to create massive numbers of packing possibilities (very unlikely imo).

    You can replace UPX and Morphine with any other packer or crypter. Above two were meant just as an example. Or are there any limitations in multipacking samples and order of packers/crypters used.

    I'm open for any corrections regarding this "theory" from AV experts (like Stefan,Marcos or IBK...)
     
  23. gigaman

    gigaman Guest

    I don't think it's possible to substitute unpacking capabilities by adding artificially created packed signatures. Many packers (e.g. Morphine, Yoda, etc.) use some kind of random encryption - so if you pack a file twice with the same packer, you'll get different results. It's not possible to generate the signatures in advance for such packers.
    And even for the "stable packers" (that generate always the same result for the same input file), it's often enough to change one insignificant byte at the beginning of the file, and the packed result is completely different again.
     
  24. Firefighter

    Firefighter Registered Member

    Joined:
    Oct 28, 2002
    Posts:
    1,670
    Location:
    Finland
    Thanks for an advice. I only thought that the web was boundless, but if that has national/local borders, after that I'm speechless. ;)

    I have to admit that some av:s are difficult to configure right. :cool:

    Best regards,
    Firefighter!
     
  25. GuruGuy

    GuruGuy Registered Member

    Joined:
    Jun 18, 2005
    Posts:
    48
    All of this analysis and what's been forgotten is the fact that each AV analyzed the SAME FILES.

    Some AV's did better than others, some did miserably.
     
Loading...
Thread Status:
Not open for further replies.
  1. This site uses cookies to help personalise content, tailor your experience and to keep you logged in if you register.
    By continuing to use this site, you are consenting to our use of cookies.