About the "product vs 21895 malware samples" threads

Discussion in 'other anti-virus software' started by LowWaterMark, Apr 13, 2011.

Thread Status:
Not open for further replies.
  1. LowWaterMark

    LowWaterMark Administrator

    Joined:
    Aug 10, 2002
    Posts:
    18,278
    Location:
    New England
    I have removed those threads for the following reasons...

    First, a person can't post what is presented as a legitimate and valid test, but, when challenged as to the flaws in test methodology, fall back to saying they are not a professional tester and these were done for fun. The methodology flaws remain and that slight qualifier somewhere down in the thread won't stop the majority of readers from taking the results as some how meaningful when they are not.

    Second, when you "collect lots of samples" and call them all malware, yet, have not verified the samples, your testbed is invalid. You can't just upload some samples to VT (or use a local product's scanner), to declare the samples legitimate malware for testing against the rest. Also, you can not collect samples for 10 days, test on the 10th day, and still call the samples 0-day.

    Without verifying and/or testing the individual malware samples, you can't know how many of the samples are real malware versus harmless or broken files. With large numbers of collected samples, it is highly likely that the sample set has many duplicates, perhaps only named differently. Professional organizations always de-dupe their malware sets to prevent detection results from being skewed.

    If a collection contains a wide mix of malware types, including those "potentially unwanted..." items, like riskware or similar, you need to separate them into groups for testing since listing total detections counts versus misses is skewed and misleading for the actual severity involved.

    These are just a few of the reasons why we view these types of home grown tests as both meaningless and yes, even damaging since some people will start to blindly follow the results as an indicator of product effectiveness.

    In any case, I am not actually trying to define proper testing procedures here. I'll leave that to groups like AMTSO. What I will say is that we do not encourage anyone to start collecting malware samples and then when they have "enough" start running tests and publishing their results. If you want to become a tester, then become the real thing. Don't do a half measure of testing without the needed expertise, proper tools and procedures. It does not prove anything at all regarding the products tested, and it misleads those who read your results.

    This thread will be added as an extension of the posting policy which disallows that whole YouTester category of malware tests.
     
Thread Status:
Not open for further replies.
  1. This site uses cookies to help personalise content, tailor your experience and to keep you logged in if you register.
    By continuing to use this site, you are consenting to our use of cookies.