Malware Research Group #23 Test

Discussion in 'other anti-malware software' started by Dragons Forever, Apr 15, 2010.

Thread Status:
Not open for further replies.
  1. http://malwareresearchgroup.com/?page_id=2

    You can download the test on that page. Here are the results anyway...


    ~~ image of results table removed ~~

    ~~ The copyright within the PDF document is very specific about not publishing or reproducing its contents. ~~

     
    Last edited by a moderator: Apr 15, 2010
  2. ALookingInView

    ALookingInView Registered Member

    Joined:
    Sep 14, 2009
    Posts:
    365
    Feel lucky you didn't decide to post this over at ___ (another forum). o_O

    Anyway, only one major surprise to me, everything else looks about right (whether they're credible or not the results are believable IMO).
    ...well ..with the exception of a certain one anyway, and no I don't/won't use it.
     
  3. vojta

    vojta Registered Member

    Joined:
    Feb 26, 2010
    Posts:
    830
    Crystal clear. You won't be banned for copyright reasons, that's for sure.:blink:
     
  4. subset

    subset Registered Member

    Joined:
    Nov 17, 2007
    Posts:
    825
    Location:
    Austria
    There is a line in the pdf which shows some lack of knowledge.
    "... Oasis components are not used in static scanning."
    That's wrong. Oasis is used by OA for static scanning.
    And as the tests were performed with Internet access, OA used Oasis for sure.

    There may be no difference related to the scan results, if only malware is scanned, but OA uses its own whitelist during static scanning to prevent FPs by the AV engine, in fact Oasis overrules AV detections.

    Cheers
     
  5. Sveta MRG

    Sveta MRG Registered Member

    Joined:
    Aug 16, 2009
    Posts:
    207
    We are totally aware of that, but as NO false positives were used in this test it made no difference.

    Internet access was active yes, but all programs were tested at the same time and giving them exactly the same testing conditions was necessary.

    Regards,

    Sveta
     
  6. dlimanov

    dlimanov Registered Member

    Joined:
    Jun 10, 2009
    Posts:
    204
    Hi Sveta,
    Why wasn't Hitman used in this test? Its sole function is an on-deman scanning, I would think it would be very fitting among the products you tested.
    Thanks!
     
  7. Sveta MRG

    Sveta MRG Registered Member

    Joined:
    Aug 16, 2009
    Posts:
    207
    Testing cloud based Anti-Malware applications requires a more complex methodology.

    By their very nature, many cloud based applications are more intelligent and can become aware they are being tested. For instance, if we were to scan 250K samples with uniform file names MRG_TEST_01 etc sitting in a folder on the desktop of a single system a vendor could be aware we are testing malware!

    We have devised a methodology for countering any such issues, but details of this are private – but, we can say that it would not have been compatible with this test.

    Regards,
    Sveta
     
  8. kjdemuth

    kjdemuth Registered Member

    Joined:
    Jul 29, 2005
    Posts:
    2,960
    Location:
    Boston, MA
    MRG,
    How does your test differ from AV-comparitives and VB100? Asquared received a less than stellar performance lately on I believe the VB100. On your test it received top honors.
     
  9. CogitoTesting

    CogitoTesting Registered Member

    Joined:
    Jul 4, 2009
    Posts:
    901
    Location:
    Sea of Tranquility, Luna
    I think that MRG did not test for false positive and both VB100 and AV-Comparatives do.
     
  10. Jeroen1000

    Jeroen1000 Registered Member

    Joined:
    Aug 18, 2008
    Posts:
    162
    Bit surprisded here too. G-Data and Avira near the top. Not surprising. Only A-squared is. I would also love to know how PrevX and Hitman Pro would do on this test. Aren't Mcaffee and Norton in the cloud too?
     
  11. Sveta MRG

    Sveta MRG Registered Member

    Joined:
    Aug 16, 2009
    Posts:
    207
    Well as far as I know AVC didn't test A-Squared and they failed only one part of VB100 test. Please read the VB100 report as see where exactly A-Squared/Ikarus failed.

    Our On Demand tests are strictly about detection, we do not use false positives, therefore we give out awards to those with highest detection rate.
     
  12. kjdemuth

    kjdemuth Registered Member

    Joined:
    Jul 29, 2005
    Posts:
    2,960
    Location:
    Boston, MA
    Thanks for the quick reply.
     
  13. abels

    abels Registered Member

    Joined:
    Apr 14, 2007
    Posts:
    102
    Location:
    Danang, VN
    How does a2 anti-malware have a highest detection rate? even higher than G-Data (Bitdefender and Avast engine) and Avira (a lot of users).
     
  14. Matthijs5nl

    Matthijs5nl Guest

    A note: FP's didn't count in this test.

    Reasons:
    1. A-Squared also uses two engines (Emsisoft and Ikarus)
    2. A-Squared has an awful lot of signatures, look at the signature updates.
    3. Ikarus has really aggresive heuristics, and because FP's don't count in this test it will get a high detection rate.
     
  15. bellgamin

    bellgamin Very Frequent Poster

    Joined:
    Aug 1, 2002
    Posts:
    5,648
    Location:
    Hawaii
    So would I. Prevx declines to be tested. They say that tests are NG when it comes to Prevx. Hmmmmm........ :cautious: :shifty: ;) :cautious:
     
  16. progress

    progress Guest

    So the "Flag-all-files-AV" is the winner, I see ... :rolleyes:
     
  17. Sveta MRG

    Sveta MRG Registered Member

    Joined:
    Aug 16, 2009
    Posts:
    207
    No, "detect-most-files-AV" is the winner, all files malicious;)
     
  18. doktornotor

    doktornotor Registered Member

    Joined:
    Jul 19, 2008
    Posts:
    2,047
    Yes, so if I write an "AV" that will flag all files as malicious then I surely win, right? Sounds like your methodology is kinda broken.
     
  19. Sveta MRG

    Sveta MRG Registered Member

    Joined:
    Aug 16, 2009
    Posts:
    207
    Flagging files is one thing, detecting verified samples is another. Don't mix those two things, I was very clear in my previous post.
     
  20. progress

    progress Guest

    Yes - Doktornotor AV will get the golden award :D
     
  21. doktornotor

    doktornotor Registered Member

    Joined:
    Jul 19, 2008
    Posts:
    2,047
    How's it different in your case? You say you only test w/ malware. So, if I flag everything, I can't miss anything and I win. Of course such AV is totally useless, but definitely a winner.
     
  22. Sveta MRG

    Sveta MRG Registered Member

    Joined:
    Aug 16, 2009
    Posts:
    207
    Well very different mate, you can flag whatever you want and detect it, but that doesn't mean that you product is good.
    I my book its simple, 250+k of samples (real malware), the application that detects the most, wins.
     
  23. doktornotor

    doktornotor Registered Member

    Joined:
    Jul 19, 2008
    Posts:
    2,047
    Well, no offense - but, logic wasn't your forte back at school, right? Once again, the test looks heavily broken, since it doesn't require any real intelligence to pass it. You only need one and exactly one universal signature which matches all files. And yeah, it doesn't mean the product is good - then, why are you using such flawed methodology?
     
  24. sded

    sded Registered Member

    Joined:
    Jun 4, 2004
    Posts:
    512
    Location:
    San Diego CA
    Sort of like current HIPS tests. You sit a user in front of a computer and tell him that all the popups will be malware. Fails must be because the application doesn't generate quite enough popups, so you have quiet modes for users and robust modes for testing. I think we did that with monkeys and food when I was in school. :)
     
  25. Sveta MRG

    Sveta MRG Registered Member

    Joined:
    Aug 16, 2009
    Posts:
    207
    The methodology is not flawed;) And how are you going to create one universal signature to detect various types of malware? I would really like to know, as well as the whole internet security industry for that matter...
     
Loading...
Thread Status:
Not open for further replies.