How Reliable and Accurate is this?

Discussion in 'other anti-virus software' started by PJC, Dec 1, 2012.

Thread Status:
Not open for further replies.
  1. PJC

    PJC Very Frequent Poster

    Joined:
    Feb 17, 2010
    Posts:
    2,959
    Location:
    Internet
  2. King Grub

    King Grub Registered Member

    Joined:
    Sep 12, 2006
    Posts:
    814
    Out of those particular 0-day samples that day and using on-demand scanning only as a test method, probably completely.
     
  3. Firecat

    Firecat Registered Member

    Joined:
    Jan 2, 2005
    Posts:
    7,927
    Location:
    The land of no identity :D
    This doesn't take into account the behavioural protection systems employed in all these products. Thus, by default it seems to favour multi-engine products and those products which have more frequent database updates. It might be one way of looking at it but this isn't all that indicative of zero-day protection since behavioural systems do count towards protection at zero day....(as well as malware URL blockers/download protection/reputation based queries, etc.)
     
  4. kjdemuth

    kjdemuth Registered Member

    Joined:
    Jul 29, 2005
    Posts:
    2,960
    Location:
    Boston, MA
    Malwarebytes at 3% and Emsisoft at 41%? Sounds kind of suspect.
     
  5. Macstorm

    Macstorm Registered Member

    Joined:
    Mar 7, 2005
    Posts:
    2,531
    Location:
    Sneffels volcano
    To me, seems to be a sponsored website trying to reverse the disastrous image left by certain vendor on the other two big tests.
     
  6. Cudni

    Cudni Global Moderator

    Joined:
    May 24, 2009
    Posts:
    6,956
    Location:
    Somethingshire
  7. Macstorm

    Macstorm Registered Member

    Joined:
    Mar 7, 2005
    Posts:
    2,531
    Location:
    Sneffels volcano
    Even more suspicious yet, the "certain vendor" does not appear in the list. Hard to believe on it.
    Payback will be on the way: https://threatcenter.crdf.fr/?Stats

    Sorry, their testing results doesn't make any sense to me.
     
  8. Firecat

    Firecat Registered Member

    Joined:
    Jan 2, 2005
    Posts:
    7,927
    Location:
    The land of no identity :D
    First of, it seems all products are not updated at exactly the same time. For this reason the results are already somewhat skewed.

    Secondly, the samples are collected from their database. Now their database is based on community submissions and from VirusTotal. We do not know for e.g. how many of these are simply executables packed with some fancy packer which triggers packer detection from some AVs (note that quite a few of the AVs scoring well do in fact detect packers under specific circumstances).

    I'm not sure how valid the test is, because it's based on one vendor's database instead of an industry standard, vendor neutral malware collection. It might actually be a "honest" test but something tells me it's skewed from the start due to reasons I have mentioned earlier.
     
  9. Disney

    Disney Registered Member

    Joined:
    Oct 15, 2012
    Posts:
    103
    Location:
    USA
    Grub is 100 % on this . The test done is probably very accurate . Question is how good was the test to accurately display how good the software is overall in real world . You need not go far to realize Eset and Fortinet are nowhere near 1 and 2 as far as overall protection .
     
  10. hamlet

    hamlet Registered Member

    Joined:
    May 10, 2005
    Posts:
    201
    I don't think it is that far fetched to think the ESET would be at or near the top in detecting zero day threats which is what this test seems to measure. NOD32 has long been a leader in proactive heuristic detections. I think that for some years it has done very well in that category at AV Comparatives.

    I must say, however, it is a bit surprising to see MBAM near the bottom of the results. That does not jive with any other evidence I have seen on that product.
     
  11. trjam

    trjam Registered Member

    Joined:
    Aug 18, 2006
    Posts:
    9,057
    Location:
    North Carolina
    I think it is totally accurate. For those of you who have bailed on Eset long ago have alot to learn. Looking down this list looks about right.
    MBAM gets most of its samples from forum members so its score on zero day is not surprising.
     
  12. hamlet

    hamlet Registered Member

    Joined:
    May 10, 2005
    Posts:
    201
    I did not know that about MBAM. I have been running it real time for a month or two and it really has not detected anything other than when I test my setup at various sites. I first started using it because it seemed like the tool of choice for support agents from a variety of other malware companies.
     
  13. Ibrad

    Ibrad Registered Member

    Joined:
    Dec 8, 2009
    Posts:
    1,949
    Guys I have worked with this site the people I believe seem trustworthy. These results are on-demand VT results as they test them. Looking over the results they seem pretty accurate from pure VT results. I would submit a lot of malware to them in the past and the results never seemed off to me.
     
  14. ctrlaltdelete

    ctrlaltdelete Registered Member

    Joined:
    Oct 16, 2005
    Posts:
    318
    Location:
    NL
    Doesn't seem accurate to me.

    Emsisoft uses the Bitdefender engine and their own engine.
    Can't see that in the stats...
     
  15. VectorFool

    VectorFool Registered Member

    Joined:
    Oct 21, 2012
    Posts:
    280
    Location:
    India
    Using the same engine does not mean they will have the same(or comparable results) since,
    1)Heuristics have been toned down in Emsisoft to favor faster scanning and low memory footprint
    2)Emsisoft does not use the Bitdefender engine on real time protection, they use a much lighter Mamutu

    there are many products like Estsoft Alylac and BullGuard which use BD engine but don't display equivalence to the detection and removal rate standards set by BD, but run much much lighter than BD.
    There is a difference between implementing the engine and a simple rebrand :)
     
  16. ctrlaltdelete

    ctrlaltdelete Registered Member

    Joined:
    Oct 16, 2005
    Posts:
    318
    Location:
    NL
    1) Heuristics have not been toned down, just checked with a big sample pack.
    2) Not relevant to this topic, it's about on-demand scans. But i'd like to inform you that the double engine is used for realtime protection (check it with a simple eicar.com test file) and in addition there's the Behavior Blocker a.k.a. Mamutu.

    Heuristics are also enabled on Virustotal. Therefore i say that the stats shown on the cdrf site are not accurate.
    If i'm sure they're not accurate for 1 product, i don't care about the other results in the same test. And that's what this thread is all about.
     
  17. PJC

    PJC Very Frequent Poster

    Joined:
    Feb 17, 2010
    Posts:
    2,959
    Location:
    Internet
    Now, results are slightly changed...
     

    Attached Files:

    • 1.jpg
      1.jpg
      File size:
      90.2 KB
      Views:
      1,282
  18. SLE

    SLE Registered Member

    Joined:
    Jun 30, 2011
    Posts:
    361
    You don't really believe this sentence, or? :ninja:
    _
    The CRDF stats are valid for on-demand results of their samples, from their sources which not necessary have to do something with real word dissemination.
     
  19. Ibrad

    Ibrad Registered Member

    Joined:
    Dec 8, 2009
    Posts:
    1,949
    Has Emsisoft upgraded their engine off the Ikraus engine on VT to the Bitdefender one? It been a while since I uploaded something and compared detection of the two.
     
  20. andyman35

    andyman35 Registered Member

    Joined:
    Nov 2, 2007
    Posts:
    2,336
    For me,the main point of interest in that table is that the best product still missed over a third of the samples,highlighting again the limitations of blacklisting.
     
  21. xxJackxx

    xxJackxx Registered Member

    Joined:
    Oct 23, 2008
    Posts:
    4,049
    Location:
    USA
    Some of us bailed for quality issues, not detection rates.
     
  22. ctrlaltdelete

    ctrlaltdelete Registered Member

    Joined:
    Oct 16, 2005
    Posts:
    318
    Location:
    NL

    Yes, they did.
     
  23. Noob

    Noob Registered Member

    Joined:
    Nov 6, 2009
    Posts:
    6,468
    +1 ctraltdelete
    I feel EAM should perform better since it's an On Demand test and it's using the same BD engine but at the end whatever, im fine with those results. :D
     
  24. spywar

    spywar Registered Member

    Joined:
    Oct 23, 2012
    Posts:
    583
    Location:
    Paris
    I have some additional access to their database, there are some FPs and multiple threats so the results are not 100 % reliable ...
     
  25. Solarlynx

    Solarlynx Registered Member

    Joined:
    Jun 25, 2011
    Posts:
    1,915
    I am not surprised with MBAM results. In some thread at Wilders' (sorry, I couldn't find it) I read that MBAM is aimed not to substitute a regular av, but to make some work other avs don't.
     
Loading...
Thread Status:
Not open for further replies.