Disparities In AV Lab Testing

Discussion in 'other anti-virus software' started by itman, Sep 18, 2018.

  1. itman

    itman Registered Member

    Joined:
    Jun 22, 2010
    Posts:
    8,593
    Location:
    U.S.A.
    In the most recent AV-Comparatives Realtime test: https://www.av-comparatives.org/com...=2018&chart_month=8&chart_sort=0&chart_zoom=0 , McAfee scored 100% in detection.

    In the most recent Malware Research Group corresponding 360 test: https://www.mrg-effitas.com/wp-content/uploads/2018/08/MRG-Effitas-2018Q2-360-Assessment.pdf , McAfee scored dead last in protection capability.

    I couldn't think of a better example to state that AV Lab test results widely vary and no one test should be used as determination in an AV product's protection capability.
     
  2. guest

    guest Guest

    Until all labs decide to use the same methodology with the same samples as a "baseline" test, then maybe, using their own methodology as secondary test; i will never trust any labs, even some "home-testers" do better than them.
     
  3. bellgamin

    bellgamin Registered Member

    Joined:
    Aug 1, 2002
    Posts:
    8,102
    Location:
    Hawaii
    Is McAfee a super great AV? Well... yes & no. :confused:
     
  4. IBK

    IBK AV Expert

    Joined:
    Dec 22, 2003
    Posts:
    1,886
    Location:
    Innsbruck (Austria)
    Different products, different versions, and most importantly completely different time of testing (in this given example MRG in Q2 [test finished in June] while AV-C tested in August). Nevertheless, I also always recommend to look at various lab / tests /methods results to get a bigger picture.
     
  5. EASTER

    EASTER Registered Member

    Joined:
    Jul 28, 2007
    Posts:
    11,126
    Location:
    U.S.A. (South)
    Including yourself as "home tester"? Pls confirm with yes? With the ambition from your extreme defensive measures I can only assume you have done your own fair share of testing in comparison to measure results from labs to real-time local tests.
     
  6. mekelek

    mekelek Registered Member

    Joined:
    May 5, 2017
    Posts:
    518
    Location:
    Hungary
    mediocre, not as bad as for example Panda but not in top10 either
    weak against non-PE but at least has signatures faster than a dead snail, aka Panda.
     
  7. guest

    guest Guest

    Indeed, i did in the past, i had a dedicated machine (not a VM) for realistic results and real 0-days samples which were manually downloaded (or USB transferred) then executed on the test machine ( as a normal user would do).
    But since my security setup block is made to block all execution of files located in commonly known landing areas, i don't even bother doing tests anymore.
    Those kind of malware tests are good for evaluating AVs, but they are useless against deny-default setups, can't execute, can't deliver payloads, can't infect.
    The only attack vectors i can see that defeat default-deny setups are network attacks (SMB exploits, etc...) or compromised legit software installers (Ccleaner style).
     
    Last edited by a moderator: Sep 19, 2018
  8. itman

    itman Registered Member

    Joined:
    Jun 22, 2010
    Posts:
    8,593
    Location:
    U.S.A.
    Expounding on AV-C's previous comments, AV-C tested McAfee Internet Security 21.2. MRG test Total Protection 16.0 R13. I gave up trying to get the latest release number for each product. I could not find this info on the McAfee web site or anywhere for that matter.

    Assumed is Total Protection has the same base malware protections Internet Security has with additional add-on security features; password manager, etc. for example.

    McAfee's improved scoring on the latest AV-C test is most likely a result of correcting security deficiencies found in the earlier MRG test.

    The main criteria I use in assessing AV protection capability for a product is consistent scoring across all AV lab tests the vendor participates in. If there is a wide deviation in scores, it can be assumed that there are base security protection issues with the product that are being corrected via on-the-fly testing.
     
    Last edited: Sep 19, 2018
  9. Gein

    Gein Registered Member

    Joined:
    Dec 8, 2013
    Posts:
    219
    It could just be a bug with mcafee. Like the one that webroot had in a previous MRG test, and I think emsisoft during one of the av-c test.
     
  10. itman

    itman Registered Member

    Joined:
    Jun 22, 2010
    Posts:
    8,593
    Location:
    U.S.A.
    Appears to be not the case as far as MRG 360 tests go. I went though the last 4 quarterly reports. Below are the results:

    Q1-2018 - 3rd lowest ranking
    Q4-2017 - 2nd lowest ranking
    Q3-2017 - lowest ranking
    Q2-2017 - 2nd lowest ranking

    So as far as McAfee goes in this testing series, they have been a consistent poor performer.

    What would of interest is if McAfee always submits to MRG first, prior to testing by any other major AV labs like AV-C, AV-Test, etc.. In other words, "take the hit" from a lessor known and therefore less publicly referenced lab. Fix the deficiencies. Then move on to the major lab players which get broader public exposure.
     
    Last edited: Sep 20, 2018
  11. Gein

    Gein Registered Member

    Joined:
    Dec 8, 2013
    Posts:
    219
  12. Firecat

    Firecat Registered Member

    Joined:
    Jan 2, 2005
    Posts:
    8,251
    Location:
    The land of no identity :D
    Didn't McAfee transition to an engine that has few signatures and is effectively cloud dependent?
     
  13. mekelek

    mekelek Registered Member

    Joined:
    May 5, 2017
    Posts:
    518
    Location:
    Hungary
    i call cloud/ML/AI detections as signatures as well. suprisingly McAfee's ML engine is one of the first on Virustotal to sig something.
     
  14. Firecat

    Firecat Registered Member

    Joined:
    Jan 2, 2005
    Posts:
    8,251
    Location:
    The land of no identity :D
    I have had that engine fail so many times, I just can't trust McAfee anymore. I don't know if it improved in the last two years, but the new engine was an effective downgrade and left many computers infected a couple of years ago.
     
  1. This site uses cookies to help personalise content, tailor your experience and to keep you logged in if you register.
    By continuing to use this site, you are consenting to our use of cookies.