The New Anti-Virus Test - Packers Support

Discussion in 'other anti-virus software' started by IlyaOS, Oct 9, 2006.

Thread Status:
Not open for further replies.
  1. Inspector Clouseau

    Inspector Clouseau AV Expert

    Joined:
    Apr 2, 2006
    Posts:
    1,329
    Location:
    Maidenhead, UK
    I can assure that Stefan has the required level of technical professionalism to make such comments.
     
  2. BlueZannetti

    BlueZannetti Registered Member

    Joined:
    Oct 19, 2003
    Posts:
    6,590
    To all:

    1. Let's keep the discussion technically oriented.

    2. When an acknowledged expert (or two) in a field points out potential experimental limitations of an approach that I'm following, I usually step back to consider the points that they are making and adjust accordingly; even if I am expert in the same field. That would be a recommendation that I would stongly encourage in this case.

    Blue
     
  3. Inspector Clouseau

    Inspector Clouseau AV Expert

    Joined:
    Apr 2, 2006
    Posts:
    1,329
    Location:
    Maidenhead, UK
    ...and that makes you a valuable person and separates you from pumpkins. And to be honest i didn't expect anything else from you :D
     
  4. BlueZannetti

    BlueZannetti Registered Member

    Joined:
    Oct 19, 2003
    Posts:
    6,590
    Well thank-you!

    By the way, I really like pumpkins - especially in pumpkin pie! Yumm!

    Blue
     
  5. Inspector Clouseau

    Inspector Clouseau AV Expert

    Joined:
    Apr 2, 2006
    Posts:
    1,329
    Location:
    Maidenhead, UK
    Difference is here that those pumpkins are innocent and don't mind if you hold them down under cold water for a few minits...
     
  6. IBK

    IBK AV Expert

    Joined:
    Dec 22, 2003
    Posts:
    1,875
    Location:
    Innsbruck (Austria)
    I did not want to reply anything to this thread/test before and look how it evolves, but yes, my thoughts are similar to what Stefan and Mike said (and it is nice from them to explain it so exhaustive) - what they say are facts and can be believed (if they do not know it, who else?) :p
     
  7. Inspector Clouseau

    Inspector Clouseau AV Expert

    Joined:
    Apr 2, 2006
    Posts:
    1,329
    Location:
    Maidenhead, UK
  8. Inspector Clouseau

    Inspector Clouseau AV Expert

    Joined:
    Apr 2, 2006
    Posts:
    1,329
    Location:
    Maidenhead, UK
    And by the way why is ESET behind Bitdefender? They do FOR SURE more unpack than Bitdefender! Anyway off to VB now have a nice discussion here and please keep it technical :)
     
  9. Stefan Kurtzhals

    Stefan Kurtzhals AV Expert

    Joined:
    Sep 30, 2003
    Posts:
    702
    I made that comment because there are some inconsistencies which at once made me think that. When you are in the AV business for a long time, you get a feeling for that. As Mike said, why was ESET rated so strange? Why were heuristic detections not rated for other products? Seems the programs were rated with different measures.

    The funny thing is, KAV doesn't need anyone to publish "tests" like this one. They have an excellent, top-rate scan engine, there is no doubt about it. So do have others.
     
  10. Inspector Clouseau

    Inspector Clouseau AV Expert

    Joined:
    Apr 2, 2006
    Posts:
    1,329
    Location:
    Maidenhead, UK
    Yub. Besides it might do even business damage for the better scoring products in such tests if you are listed in such amateurish tests. And yes of course i agree with Stefan that there is no need to prove once again in an out of question "test" that KAV has a solid av engine. There's absolutely no doubt about it. But spoiling other av companies work (as you might notice F-Prot was not even included so i defend here basically our own "opponents") without knowing how to perform this test correct is just childish and amateurish. This should give to think about :rolleyes:
     
  11. IBK

    IBK AV Expert

    Joined:
    Dec 22, 2003
    Posts:
    1,875
    Location:
    Innsbruck (Austria)
    well, i meant if not those that work closely to av developing and know how the products works in depth :D
     
  12. Firecat

    Firecat Registered Member

    Joined:
    Jan 2, 2005
    Posts:
    8,225
    Location:
    The land of no identity :D
    I'm not sure so sure that Eset unpacks more than BitDefender. It may have been true in the past but now that BD has generic unpacker as well as VB emulator, hmm :doubt:

    As for commenting on this test, I completely trust the Inspector's comments. The method of testing was not optimal IMO. :)
     
  13. hbkh

    hbkh Registered Member

    Joined:
    Jan 15, 2004
    Posts:
    129
    Location:
    Ohio, USA
    .ru is Russian not Romainian (.ro is Romania) :)
     
  14. Seishin

    Seishin Registered Member

    Joined:
    Aug 2, 2006
    Posts:
    204
    I realised now about my mistake. Spent too many hours in front of this box and misplaced the "u" for an "o". Damn! Sorry.

    :p
     
  15. IlyaOS

    IlyaOS Registered Member

    Joined:
    Nov 13, 2005
    Posts:
    29
    Inspector, i know, you are more advanced in technogy aspects then me and i don't wanna dispute. But explain me just two thinks:

    1. Why KAV and some other AV can detect packed samples without any problems? They have unique technology? :)

    2. It was our first test of packers support, so we'll correct methodology and your expert's assistance would be very useful. What we have to change from you point of view to get right, appropriate results.

    Anyway thanks for critic even if i'm not agree with it.

    To everybody: please post all your questions, suggestions and feelings about this test, your opinion is very important!
     
  16. Firecat

    Firecat Registered Member

    Joined:
    Jan 2, 2005
    Posts:
    8,225
    Location:
    The land of no identity :D
    No, its because these companies spent some time in optimizing their unpack engine to support more types of packers. :)

    I'm not sure that such a test can be reliably performed as there is no way to determine whether any vendor has specifically added added detection for packed variants of the malware used for the test (unless of course you start decompiling the database and looking at the signatures, but that is illegal). :doubt:

    EDIT: Maybe one thing you can try is using the absolute latest version of the packers to pack your infected files. There would be a high probability that the vendors using specific signatures for packed variants of malware may not have updated the database to detect the file packed with the new version of the packer. This wouldn't be a problem for AVs with good unpack engines however. But even this method probably has flaws and is not a sure-shot fix.
     
  17. RejZoR

    RejZoR Lurker

    Joined:
    May 31, 2004
    Posts:
    6,426
    Yeah, i have some doubts about superiority of NOD32 over BitDefender 10.
    BD10 engine is brutal in its latest stages, thats a fact. Maybe it still lags in terms of speed but unpacking is certanly on very very high level.
     
  18. marcromero

    marcromero Guest

    As far as av tests go, I view all of them with skepticism, even though the av product I currently use scored well on your tests, it seems your test dethroned some of the scared cows, then the demi-gods weighed in (security experts) and the first thing to be brought into question is testing methodology, then the flock follows, this is the standard scenario in security newsgroups with any new av tests by an unknown. On a more personal note, your tests were interesting, flawed or not. I really don't know, only what I read in the forums...
     
  19. Tweakie

    Tweakie Registered Member

    Joined:
    Feb 28, 2004
    Posts:
    90
    Location:
    E.U.
    IlyaOS,

    IMHO, the questions you might want to answer when testing how AVs handle (re)packed samples are the following (I'm summing up some of the arguments developed by other contributors):

    1/ Does the antivirus provide a specific/static unpacker for a given packer ?
    . Depends on the packer

    2/ Are the malware signatures picked in such a way that unpacked malwares can be detected by signature ?
    a. Depends on the malware (different signatures use different references)
    b. Depends on the packer (the structure of the sample after unpacking may be different from its structure before packing).


    3/ Are genuine files (e.g. notepad.exe) flagged as suspicious when packed with given packer ?
    . Depends on the packer

    4/ Is the antivirus able to emulate through the whole unpacking code (can be useful for generic unpacking and might be necessary for heuristics detection) ?
    . Depends on the packer

    5/ When a sample is found packed ITW, does the AV also provide a signature for the unpacked sample ?
    . Depends on the malware and on the packer


    In your test, you tried (and, I think, failed) to answer the question 1/ without taking into account answers to questions 2->5.

    Question 2:
    + For answering question 2b, you need to understand how the packer operates, either by reverse engineering it or by comparing the initial sample with the executable obtained after manual unpacking.
    + Answering question 2a, is difficult: you would need to use a packer that does significantly alter the file structure and that the AV claims to support as a reference and throw a large number of packed malwares at it --> the answer to question 2 is not a simple yes/no, but a percentage per packer. And if you want this percentage to be meaningful, you need to use a large test bed and/or to chose carefully your samples.

    Question 3:
    Easy test. Such a behavior (flagging every packed samples) is to be considered as bad for a desktop AV, excepted if the packer's author specifically claims that it has been conceived to fool AVs.

    Question 4:
    Difficult test although AVs like bitdefender or Norman could be tested by checking for a "BehavesLike" or "Sandbox" detection.

    Question 5:
    Needs access to a good collection of ITW malwares, and real understanding on what/how to unpack (see question 2).

    Conclusion: testing how an AV support runtime packers without reverse-engineering it is not an easy task. I wish you good luck for your next test(s).
     
  20. Devil's Advocate

    Devil's Advocate Registered Member

    Joined:
    Feb 5, 2006
    Posts:
    549
    Thank you Tweakie for your summary of the issues.

    As always a good instructional post on Anti-viruses and testing.
     
  21. Graystoke

    Graystoke Registered Member

    Joined:
    Aug 15, 2003
    Posts:
    1,506
    Location:
    The San Joaquin Valley, California
    This is for Stefan Kurtzhols. I tried to send you a IM, but your box is full. Is there any other way I can contact you. I have a question about AntiVir Premium, and I don't want to hijack this thread.
     
  22. Stefan Kurtzhals

    Stefan Kurtzhals AV Expert

    Joined:
    Sep 30, 2003
    Posts:
    702
    Oops, cleaned up the inbox. :)
     
  23. Graystoke

    Graystoke Registered Member

    Joined:
    Aug 15, 2003
    Posts:
    1,506
    Location:
    The San Joaquin Valley, California
    Thank you Stefan. I just sent you an IM.
     
  24. IlyaOS

    IlyaOS Registered Member

    Joined:
    Nov 13, 2005
    Posts:
    29
    Tweakie and Firecat: Thank you, these're very useful comments for us.
     
Thread Status:
Not open for further replies.
  1. This site uses cookies to help personalise content, tailor your experience and to keep you logged in if you register.
    By continuing to use this site, you are consenting to our use of cookies.