Anti-Malware Testing and Ratings

Discussion in 'other anti-malware software' started by greyowl, Dec 8, 2009.

Thread Status:
Not open for further replies.
  1. greyowl

    greyowl Registered Member

    Joined:
    Feb 14, 2008
    Posts:
    85
    I am wondering if there is any third party testing and rating of anti-spyware apps which is similar to AV-Comparative.
     
  2. jmonge

    jmonge Registered Member

    Joined:
    Mar 20, 2008
    Posts:
    12,883
    Location:
    Canada
  3. greyowl

    greyowl Registered Member

    Joined:
    Feb 14, 2008
    Posts:
    85
    Jmonge,

    Thanks for the reference. This is from 2004. I am looking for something that is more current.
     
  4. jmonge

    jmonge Registered Member

    Joined:
    Mar 20, 2008
    Posts:
    12,883
    Location:
    Canada
    sorry man let me find more in google:D
     
  5. greyowl

    greyowl Registered Member

    Joined:
    Feb 14, 2008
    Posts:
    85
    Thanks for the additional link.
     
  6. jmonge

    jmonge Registered Member

    Joined:
    Mar 20, 2008
    Posts:
    12,883
    Location:
    Canada
    your welcome,did you find more info?
     
  7. inka

    inka Registered Member

    Joined:
    Oct 21, 2009
    Posts:
    406
    "Sticky: Thoughts on Anti-Spyware Testing" (stickied since 2006)
    http://www.spywarewarrior.com/viewtopic.php?t=22210

    So now, late 2009, the independent 'ntinternals' guy tests, publishes his results
    see: (anti-rootkit) Hidden Process Detection (50+ products compared
    https://www.wilderssecurity.com/showthread.php?t=259876

    ...and his audience splits hairs, caught up in debating the merit in the tester's methodology and in discounting the relevance of the results. With an audience so hellbent on dismissing any (painfully) significant results, it seems reasonable to expect publication of comprehensive 'independent' test results to remain scarce.

    "The emperor has no clothes!"
     
  8. greyowl

    greyowl Registered Member

    Joined:
    Feb 14, 2008
    Posts:
    85
    Inka,

    Well said.

    If the consumer is deprived of legitimate testing results, then the antispyware companies can remain unaccountable and enjoy a marketing approach to business rather than a results approach.
     
  9. JerryM

    JerryM Registered Member

    Joined:
    Aug 31, 2003
    Posts:
    4,221
    Hi Inka,

    Boy, you nailed that. Every test that shows an application in less that tops is labeled as bad methodology or a result of paid ads in a magazine or bribes.

    Regards,
    Jerry
     
  10. jmonge

    jmonge Registered Member

    Joined:
    Mar 20, 2008
    Posts:
    12,883
    Location:
    Canada
    JerryM ans Inka this is very true in reality when it comes to money people even still or even kill for money:D my 2 pesos:D
     
  11. JerryM

    JerryM Registered Member

    Joined:
    Aug 31, 2003
    Posts:
    4,221
    Yes, and I'm glad you did not say your 2 million pesos or you might be in mortal danger.:D

    Best,
    Jerry
     
  12. jmonge

    jmonge Registered Member

    Joined:
    Mar 20, 2008
    Posts:
    12,883
    Location:
    Canada
    andale:D orale:) yes man thanks again;)
     
  13. 1000db

    1000db Registered Member

    Joined:
    Jan 9, 2009
    Posts:
    718
    Location:
    Missouri
    Unfortunately this seems to be more common than extracting and discussing the useful data. There are no "perfect" tests but most at least have some useful information and the one you were referring to I found interesting.
     
  14. nosirrah

    nosirrah Malware Fighter

    Joined:
    Aug 25, 2006
    Posts:
    561
    Location:
    Cummington MA USA
    IMO a test should go like this .

    Start with a list of infected sites and direct download URLs , no sample packs allowed . Next set up the test machine(s) with web facing software that is reasonably old (so exploits can take) . Setting the OS to its current SP with no further updates is a good way to set it a reasonable amount back (again to make exploits kick in) . Update and activate protection just before the test starts .

    Stage one . Check web protection first . Any sample that can be created by an exploit or downloaded through web protection should be collected for stage two .

    Stage two . Make a copy of the files that made it through stage one , they are needed twice . Execute each sample to determine if it is blocked by signatures/heuristics/whatever . All samples that able to execute will move on to stage three .

    Stage three . Retest the samples that are not blocked at primary execution and check for secondary detection . The majority of malware does nothing if everything from secondary execution on is blocked . Obviously this is not as good as killing it at step one or two but if an infection had no ability to ever get to its core malicious behavior then you were protected and this should on some level count .

    Stage four . For all infections that were able to completely take a cleanup should also be attempted . Success at this step should count the least but still should in a small way count . Good examples would be malware that are of the annoying type (adverts/search redirectors/settings modifiers.....) .

    An arbitrary scoring system could be 1 point for cleanup , 2 for secondary defense , 4 for primary defense and 8 for web defense for example . This would make each step twice as valuable as the step below it but the tester could come up with any fair system for step weights .

    What I like about this test model is that no app that is awful against current malware can pass and no app that is great against current malware can fail (obviously given that care was taken to gather a reasonably diverse and large list of links/sites) .


    Now compare the amount of work and skill required to do all of that to asking 50 security guys you know for samples , bunch them up in one folder and then doing a right click scan test against that folder , it is obvious why the latter is the default "test" .
     
  15. inka

    inka Registered Member

    Joined:
    Oct 21, 2009
    Posts:
    406
    Ouch!
    Nooooo... you would need to have the set of (discrete/fixed) test samples in hand!

    Many operators are clever enough to NOT serve the malware to your same IP more than once.

    Many operators are clever enough to serve the malware intermittently, rather than every visit.
    (foils blacklist maintainers followup checks to verify reported URLs)

    Many malware campaigns target specific demographics (user-agent, IP netblock, etc)
    (if you pass out a list of URLs & send various people out to test, and invite people to re-test... you should fully expect differing outcomes)

     
    Last edited: Dec 10, 2009
  16. nosirrah

    nosirrah Malware Fighter

    Joined:
    Aug 25, 2006
    Posts:
    561
    Location:
    Cummington MA USA
    There are ways around this obviously . Even using test machine in different regions would work .
     
  17. inka

    inka Registered Member

    Joined:
    Oct 21, 2009
    Posts:
    406
    sigh
    {speechless}
    aghast
     
Loading...
Thread Status:
Not open for further replies.