Interesting Emsisoft Tests

Discussion in 'other anti-malware software' started by MikeNash, Apr 30, 2009.

Thread Status:
Not open for further replies.
  1. MikeNash

    MikeNash Security Expert

    Joined:
    Jun 9, 2005
    Posts:
    1,654
    Location:
    Sydney, Australia
  2. softtouch

    softtouch Registered Member

    Joined:
    Jan 31, 2006
    Posts:
    415
    It would be interesting when it would not have been on their own website, tested by them self. Of course, when I test my own products, and place the test result on my own website, my products would be the winner...

    As I stated in another post, it detected 42 as malware on my pc, even the 42 have been all FP... so its detection rate was 100%, and other AV 0%...
     
  3. andyman35

    andyman35 Registered Member

    Joined:
    Nov 2, 2007
    Posts:
    2,336
    Interesting test results there,just a couple of points though.
    If this test was against A2 own samples how come it didn't detect 100%?
    Also testing MBAM and SAS against static samples doesn't give a true indication of their efficasy if that was how the test was performed,they're designed to detect running malware.
     
  4. MikeNash

    MikeNash Security Expert

    Joined:
    Jun 9, 2005
    Posts:
    1,654
    Location:
    Sydney, Australia
    Sure. If it's based on their own malware collection - you expect them to get 100%. But the _rest_ of the results, aside from the top spot are still interesting :)
     
  5. progress

    progress Guest

  6. MikeNash

    MikeNash Security Expert

    Joined:
    Jun 9, 2005
    Posts:
    1,654
    Location:
    Sydney, Australia
    At the bottom...

    Note

    This test was an Emsi Software internal comparison and does not represent an objective assessment of the detection performance of a-squared Anti-Malware. It stands to reason that our own in-house product will produce the best detection rate against our own Malware samples. After an in-depth analysis some files were rated as harmless in the meantime. That's why a-squared Anti-Malware did not reach 100%.
     
  7. Taliscicero

    Taliscicero Registered Member

    Joined:
    Feb 7, 2008
    Posts:
    1,439
    A-2 really should not have listed.

    Since they did add all the tested samples to there databace before testing..... its kind of like me sending 40.000 malware samples to ClamAV waiting 4 months then testing. OFC CLamAV would have 99.99% detection rate.
     
  8. andyman35

    andyman35 Registered Member

    Joined:
    Nov 2, 2007
    Posts:
    2,336
    OK thanks for answering point 1,but why test MBAM and SAS in this way that they're not designed to be used?
     
  9. MikeNash

    MikeNash Security Expert

    Joined:
    Jun 9, 2005
    Posts:
    1,654
    Location:
    Sydney, Australia

    I agree completely. The first place must be discounted - even in their own notes that say that "of course" they will come first. What's interesting for me is the rest of the pack.
     
  10. MikeNash

    MikeNash Security Expert

    Joined:
    Jun 9, 2005
    Posts:
    1,654
    Location:
    Sydney, Australia
    o_O Dunno. I guess if it's a test of scanners, you'd run a scanner. That's one for Emsisoft, not for me.
     
  11. andyman35

    andyman35 Registered Member

    Joined:
    Nov 2, 2007
    Posts:
    2,336
    Yes but a test should show a true reflection of the capabilities of the products tested in order to be credible.I put it that had these malware samples been activated on the system in question,both would have achieved a far higher percentage.Yes they're both scanners,but they're designed to scan for running malware.
     
  12. Franklin

    Franklin Registered Member

    Joined:
    May 12, 2005
    Posts:
    2,517
    Location:
    West Aussie
    Four recent samples.

    it021.exe - a-squared 4.0.0.101 2009.04.30 <- Not Detected - Win32.Malware.gen

    6007_1.exe - a-squared 4.0.0.101 2009.04.30 <- Not Detected - a variant of Win32/Tinxy.AD

    bb.jpg - a-squared 4.0.0.101 2009.04.29 <- Not Detected - Generic Dropper.cx

    setupxv.exe - a-squared 4.0.0.101 2009.04.22 <- Not Detected - virus:FraudTool.Win32.MalwareRomovalBot.b
     
  13. MikeNash

    MikeNash Security Expert

    Joined:
    Jun 9, 2005
    Posts:
    1,654
    Location:
    Sydney, Australia
    Did the other vendors catch these?
     
  14. andyman35

    andyman35 Registered Member

    Joined:
    Nov 2, 2007
    Posts:
    2,336
    If I was of a suspicious persuasion I might think that showing MBAM and SAS (2 directly competing products) in such a poor light wouldn't be bad for business at all.:rolleyes:
     
  15. Saraceno

    Saraceno Registered Member

    Joined:
    Mar 24, 2008
    Posts:
    2,404
    If a-squared came third, or last, doesn't bother me, it's only one test, so hopefully, no one will take it personally.

    And the note explains it quite well - "This test was an Emsi Software internal comparison and does not represent an objective assessment of the detection performance of a-squared Anti-Malware. It stands to reason that our own in-house product will produce the best detection rate against our own Malware samples. After an in-depth analysis some files were rated as harmless in the meantime. That's why a-squared Anti-Malware did not reach 100%."

    But just commenting on the one test - being in a two-week period, some of those AVs which aren't in the top group, may add malware signatures a few weeks later, as they might not have the staff and/or resources.

    Launching malware would be an interesting test. I still think Emsisoft would do well in launching/preventing malware, many which might be missed by an on-demand scan, as their behaviour blocking (Mamutu/a-squared anti-malware) is a core component of their software.

    But on-demand scanning, not bad - for a small company.
     
  16. Franklin

    Franklin Registered Member

    Joined:
    May 12, 2005
    Posts:
    2,517
    Location:
    West Aussie
    File it021.exe - Result: 4/40 (10.00%)

    File 6007_1.exe - Result: 11/40 (27.50%)

    File bb.jpg - Result: 10/40 (25.00%)

    File setupxv.exe - Result: 2/40 (5.00%)
     
  17. MikeNash

    MikeNash Security Expert

    Joined:
    Jun 9, 2005
    Posts:
    1,654
    Location:
    Sydney, Australia
    Typically, that's why companies publish results that show their product in a good light, and forget (or attack) those that don't.

    I don't think anyone here would be surprised at Emsisoft's motives for publishing these results.
     
  18. MikeNash

    MikeNash Security Expert

    Joined:
    Jun 9, 2005
    Posts:
    1,654
    Location:
    Sydney, Australia
    That would be a very interesting test indeed. It would be hard to do it with a high number of samples - but it would be interesting to pluck out some of the ones that few companies detected and run them through by hand to see if it made any meaningful difference to the results.
     
  19. MikeNash

    MikeNash Security Expert

    Joined:
    Jun 9, 2005
    Posts:
    1,654
    Location:
    Sydney, Australia
    Nasty.
     
  20. andyman35

    andyman35 Registered Member

    Joined:
    Nov 2, 2007
    Posts:
    2,336
    This is why the independence of the testers is of paramount importance when ascertaining the credibility of any test.
     
  21. Saraceno

    Saraceno Registered Member

    Joined:
    Mar 24, 2008
    Posts:
    2,404
    I installed some legitimate software the other night, but when I noticed it was trying to change browser settings, at least gave me an idea how a 'clean' file is behaving.

    So sometimes it's not only malware you want to prevent, you want to prevent unwanted system changes. Well to me, that's important.

    Anyway, although this is an on-demand scan, many may agree, Mamutu/emsisoft's behaviour blocker, is easy to use and provides strong protection.
     
  22. andyman35

    andyman35 Registered Member

    Joined:
    Nov 2, 2007
    Posts:
    2,336
    Agreed 100%:thumb:
     
  23. MikeNash

    MikeNash Security Expert

    Joined:
    Jun 9, 2005
    Posts:
    1,654
    Location:
    Sydney, Australia
    *grin* I couldn't agree with you more.
     
  24. Saraceno

    Saraceno Registered Member

    Joined:
    Mar 24, 2008
    Posts:
    2,404
    And same applies to Comodo, that's an extremely good result considering its a new AV. And launching the malware missed, the rest would most likely, be stopped in its tracks.
     
  25. emsisoft

    emsisoft Security Expert

    Joined:
    Mar 12, 2004
    Posts:
    312
    Location:
    Nelson, New Zealand
    When preparing this 'test' we tried to be as careful as possible to avoid arguments like "this test is fake" and "the test methods are questionable".

    It was not out intention to give an overall rating on security products. Of course we can't do that independently as a competing company.

    We just wanted to show the differences of the on demand scanners and the speed of updates. Several independent tests recently showed that the top 5 scanner detections differ only by 1%, but these tests didn't show if they missed the same 1% or different 1% of the samples.

    Another thing we wanted to uncover is the fact that antispyware tools, even if they come with "+AV" modules, can't really compete with the top antivirus players on the market.

    Regarding Malwarebytes: This freeware scanner tool (not speaking about the guard version) is highly recommended by many people (not only here), but I have not seen a single test of their scanner capabilities so far. Some independent testers I've spoken with, told me that it is not good enough to be included in their tests at all, but that's another story. Well, we were very surprised that it wasn't able to detect at least 50% of our samples. I guess most users like MBAM because of the scan speed, but it's new to me that it is made to detect only running malware. Most people use only the freeware scanner without guard imho.

    Again, our test is not a 'security' test that shows how good a program can actually protect your computer. It is a pure scan engine test. Not more, not less.

    Testing was done very carefully and reproducable. If any of the vendors needs a hash list of the missed samples, please send me a note.
     
Loading...
Thread Status:
Not open for further replies.