New Methodology Draft for PCSL Total Protection Test(Suggestions Welcome)

Discussion in 'other anti-virus software' started by pcslinfo, Feb 3, 2010.

Thread Status:
Not open for further replies.
  1. pcslinfo

    pcslinfo Registered Member

    Joined:
    Jul 18, 2008
    Posts:
    157
    Please join http://www.sectalks.com (Security Talks Blog) and give me your constructive suggestions, thank you in advance.

    Please visit http://www.sectalks.com/?p=35 to download the methodology

    Hello everyone,
    I have finished the new methodology draft for PCSL Total Protection Test after several days’ internal discussion(Special thanks to Alice@Trend Micro, Pedro@Panda and Dragos@BitDefender) and now I will release it to public for more opinions on that.

    Please give me your constructive suggestions, thank you in advance.

    Regards
    Jeffrey
    2010-02-04

    ==============================================================================================================

    About Sectalks.com
    PC Security Labs will invite the guys from security industry, research organizations, product test labs and also individual researchers to post threads in this weblog. We hope it will be a place for research experts from security industry, individual researchers and computer users to freely communicate and also show computer users how to protect yourselves with us!

    Any guys from antivirus security , research oganizations &labs or individual researchers can register and have the right to post subject on security(auther group), please email me when you finish registration and I will add you to the auther group.

    Cheers
    Jeffrey
     
    Last edited: Feb 3, 2010
  2. dawgg

    dawgg Registered Member

    Joined:
    Jun 18, 2006
    Posts:
    817
    Jeffrey, the methodology looks good, but I have some thoughts for changes...

    Sample and source collection (3) - Yes, computers which have the malware on them indicate the malware is in the wild and poses a real threat to users, although the problem with this approach is that it makes the results selective against the AV the users are using (the more popular AVs in your region), as you have selected a sample you know the AV the user is using does not detect, but other malware it does detect may have been successfully been blocked/removed.

    Eg, I help in the Kaspersky malware removal forum, and of course, sometimes malware gets through Kaspersky does not detect, so if I send samples to you, it will only be those Kaspersky misses, lowering its detection rates. The more samples I send, the lower its detections will get.
    Like your approach to using AVs to submit the samples, although that issue you have fixed by ensuring all AVs get to add an equal share of the pie.


    Also, can we have confirmation of the types of malware you use? I recall once seeing a sample on your older website/forum which was classed as adware, but of-course adware is often greyware and it is arguable whether it is malicious or not. However, I do not know if it was used your tests or not.
    I personally do not think adware and other "potentially unwanted applications/programs" should be included in malware tests.
     
  3. Pleonasm

    Pleonasm Registered Member

    Joined:
    Apr 9, 2007
    Posts:
    1,201
    Jeffrey, a few thoughts to consider:
    • I recommend that you clearly articulate the objectives you are seeking to address through the testing. Only when those are precisely defined does it make sense to pursue the methodological issues.
    • Concerning the methodology itself, it is obvious that other organizations (e.g., AMTSO, AV-Comparatives, AV-Test) have spent considerable time and effort in defining and improving their methods. It doesn’t make sense for PCSL to journey the same path, beginning at the beginning -- rather, why not build upon and improve the foundation that has already been constructed by others, if permissible?
    • Think about how the PCSL testing will incrementally add to the body of the knowledge about the performance of anti-malware products. In other words, what problems are being solved by PCSL that are not already being addressed by other testing organizations?
    I hope this guidance assists you.
     
  4. lordraiden

    lordraiden Registered Member

    Joined:
    Jan 30, 2006
    Posts:
    3,080
  5. pcslinfo

    pcslinfo Registered Member

    Joined:
    Jul 18, 2008
    Posts:
    157
    I have several approaches of sample income, like "a basket of currencies" it is the most equal way to gather balance information and sample set. To have submission from fans is also a ability for sample collection of a vendor.

    I do not use PUA for test, I have mentioned that in the methodology, please check again.

    Thank you dragg.

    What I want to solve is to simulate the real performance that one single antivirus product act on the normal pc. If you have read my former report, you will find, I have add dynamic test(I will simulate the real infection) false positive test and dynamic false positive test into a single test, and this system is already established and what I should do is perfect that, so that is why after the internal discussion I release it to public to find more suggestions.

    PCSL is a member of AMTSO, so surely in our methdology we will reflect some guidance of AMTSO

    Thank you for your suggestion Pleonasm and lordraiden
     
    Last edited: Feb 5, 2010
  6. Zombini

    Zombini Registered Member

    Joined:
    Jul 11, 2006
    Posts:
    469

    Not sure what kind of test you are trying to do, whether its static or dynamic, but if its dynamic you have clearly violated some of the principles dealing with introduction.

    "All the prevalent samples are renamed using Renamer using the format of “CRC32+.exe extension”. First, all the samples are placed into a specific folder of a system folder in virtual machine and executed one by one"

    There are three violations in this sentence itself:
    - you can't be renaming malicious files. That is not real world
    - You cannot bias the test by copying them into System32, because clearly not all malware in the wild is in that folder.
    - You should not manually copy the files onto the test machine.. you have to introduce them "naturally" through a drive-by download, email attachment, manual download etc. just like how it would happen in the wild.

    Overall, I think your testing methodology doc has ways to go before it is taken seriously by any of the AV vendors that belong to AMTSO.
     
  7. PickyPeach

    PickyPeach Registered Member

    Joined:
    Feb 4, 2010
    Posts:
    1
    I am a newbie in this particular forum and it would be great if you would reveal me the needs (or principles) that prohibit the mix of static and dynamic tests.

    a) You are right. This is not real world. It would be pretty childish to think that any laboratory tests (aka. clinic tests) can mirror the real world.

    There must be a reason why you affirm that malicious files can't be renamed. I just can't figure out which is it! Can you help? o_O o_O

    Running a file from the %System% folder has the huge advantage that it is executed with the highest UAC rights (Windows OS uses to trust and accept execution from this "blessed" path - but you certainly know this too).
    For the purpose of sample validation and to make sure that the malware has no execution restrains, it is quite usual to proceed like this.

    Hopefully you don't mind but your proposal refers to a different type of test. As per the methodology described by Jeffrey he intends to evaluate detection rate (statically and dynamically) and not protection grade.

    It seems that you are a fan of real world emulation which is really commendatory :thumb:

    Are you one of them? Tell me! What kind of tests dooes AMTSO take as seriously?

    Best regards,

    PickyPeach

    [MOVE]"Amateurs built the Ark......Professionals built the Titanic" [/MOVE]
     
  8. Zombini

    Zombini Registered Member

    Joined:
    Jul 11, 2006
    Posts:
    469
    I didn't say anything about mixing or not mixing static and dynamic tests. You can do what you want, although it has been pretty much established at this point that static tests are useless and our on their way to the museum. So if I were to test a NEW testing outfit and wanted to be taken seriously I would not be doing static tests.


    Actually, you CAN get pretty realistic in these static tests, at realistic to the point that the current crop of AV products wont be able to tell the different between real-real world and simulated-real world.

    There are plenty of heuristics that deal with file name characteristics. You mess with the file name and you mess with the detection.


    By doing so you encourage certain AV vendors (names witheld) from tuning their detections to be more aggressive with files in the %System% folder, when in reality in the real-world malware is found in equal quantity in the TIFF or in the %temp% folder.

    Unless I am missing something, the test methodology he has outlined looks more like a dynamic test, since he EXECUTES the malware. If you do that, you have to play by the rules.
     
  9. pcslinfo

    pcslinfo Registered Member

    Joined:
    Jul 18, 2008
    Posts:
    157
    If one vendor's on-demand scanner will generate a detection via distinguishing file name, I think the scanner is ridiculous. And also, the filename of malware in the real world is sometime dynamic or ramdon.

    I have not mentioned we copy them into system32, I mean system folder, %system32% is, %system% is, %temp% also is, and everytime we will change different folders to avoid someone cheating the test.
    And also, if you have read the methodology carefully, we have dynamic false positive test and static false positive test.
    These two tests are the reference objects to static malware scan test and dynamic malware block test, it is better to hold a malware scan test along with a staticfp test, also a dynamic malware block test along with a dynamic fp test. The two pairs of test will be executed under the same condition and the only difference is the sample tested, one is malware and the other is clean files. So you need care about that is any space to make a specialized rule to cheat the test to get a higher score.

    you mentioned in floor 7th :
    "Unless I am missing something, the test methodology he has outlined looks more like a dynamic test, since he EXECUTES the malware. If you do that, you have to play by the rules.
    ")
    Our dynamic test part (please note it is a part of the test) is aimed at showing how an antivirus software protects the computer from being infected, once an AV failed to detect the malware by the signature database. This test has nothing to do with how the malware gets into my computer system, or things like that. As the last proctection of the system, we only care that how AV softwares defend against an active threats.

    As mentioned above, reference test will avoid this, if they put an aggressive way to detect malware, they will get deduction in static false positive test.

    From my part, I would like to say thanks to you comments, cause any constructive comments is welcome to improve the test.
    Regards
    Jeffrey
     
    Last edited: Feb 9, 2010
Loading...
Thread Status:
Not open for further replies.