Gaming the AV-C and AV-Test Tests

Discussion in 'other anti-virus software' started by qakbot, Nov 7, 2012.

Thread Status:
Not open for further replies.
  1. qakbot

    qakbot Registered Member

    Joined:
    Aug 25, 2010
    Posts:
    380
    There is a growing concern that certain AV vendors are gaming the On-demand tests run by AV-C and AV-Tests. Does anyone have any such concerns.

    I just see way too many cases where companies are getting 100% detection, not 99.9%, not 99.6% but a full 100%.

    I'd like to understand if any of the reviewers on this board, AV-C, AV-Test can confirm this.

    There are many ways I can think of that certain vendors are gaming this test. Notably,

    1. Attempting detect the IP address range used by these reviewers and having special cloud handling for any cloud-requests coming from these machines.

    2. Detecting X positives detections in the last Y files, which puts the product in a special 'aggressive' mode.

    3. Detecting filenames and pathnames of a certain format like those used by reviews e.g. C:\FF\33\82\A9\....\FF3382A9.....exe where the FF33 is the hash of the file.

    Other similar techniques..
     
  2. RejZoR

    RejZoR Registered Member

    Joined:
    May 31, 2004
    Posts:
    6,426
    I doubt it. You have to understand that no one has reached 100% in any on-demand test. It was 99,9% but never 100% (at least as far as i'm aware of the tests). 100% was in real-world monthly tests where every file is actually executed and in this test, samples size is much smaller, meaning it's much likely for someone to score 100% than on on-demand tests with several 100.000 samples.
     
  3. Firecat

    Firecat Registered Member

    Joined:
    Jan 2, 2005
    Posts:
    7,927
    Location:
    The land of no identity :D
    Certain products do sometimes have extra large signature updates around the time of testing; so it's definitely possible. But IMO, to have resources dedicated to this is just wasting talent; AV-Test and AV-comparatives release results at different times of the year so it would mean having people track machines year round. Not exactly worth it when there's a lot of real world malware going around.

    Also, it's not a guarantee that the same internet connection is used every time, and samples are provided to the vendors only after the test. So how would they know what the file is named?
     
  4. Anth-Unit

    Anth-Unit Registered Member

    Joined:
    Oct 13, 2006
    Posts:
    108
    How does anyone ever manage to get infected when all the antivirus companies score 99.999 on all the tests?
     
  5. siketa

    siketa Registered Member

    Joined:
    Oct 25, 2012
    Posts:
    2,695
    Location:
    Zagreb, Croatia
    How about 0-day malware?
     
    Last edited: Nov 7, 2012
  6. RejZoR

    RejZoR Registered Member

    Joined:
    May 31, 2004
    Posts:
    6,426
    These are on-demand tests. In proactive tests, they usually detect far less in around 50-70%.
     
  7. vojta

    vojta Registered Member

    Joined:
    Feb 26, 2010
    Posts:
    830
    I totally believe that a good antivirus can detect 99% of yesterday's malware. People get infected with today's malware.

    Those figures only show that on demand scan is just 20% of what an antivirus should do. It's important but not enough.
     
  8. Baz_kasp

    Baz_kasp Registered Member

    Joined:
    May 1, 2008
    Posts:
    593
    Location:
    London
    Just a question for OP. If symantec got 99.9% on an AVC test, would you be suspicious of them? : )
     
  9. Firecat

    Firecat Registered Member

    Joined:
    Jan 2, 2005
    Posts:
    7,927
    Location:
    The land of no identity :D
    Sometimes, you also see the situation where a product with 99%+ score doesn't detect/stop what a lesser rated product does. Just goes to show on-demand scans aren't everything. :)
     
  10. IBK

    IBK AV Expert

    Joined:
    Dec 22, 2003
    Posts:
    1,819
    Location:
    Innsbruck (Austria)
    We look very hard to avoid gaming in the tests. We are also using methods, to avoid gaming like mentioned above and others. If you have a look at the results, no vendor has ever reached a 100% score in a file detection test performed by AVC.
     
  11. Frank the Perv

    Frank the Perv Banned

    Joined:
    Dec 16, 2005
    Posts:
    882
    Location:
    Virginia, USA
    I believe this.

    If a testing company were to lose credibility in a material and credible way, that would cost them money.

    Money is a good driver to attempt to produce a fair product.
     
  12. Taliscicero

    Taliscicero Registered Member

    Joined:
    Feb 7, 2008
    Posts:
    1,439
    Gaming these tests is most likely possible, however the people who are smart enough to figure it out are working for government or private clients and not working in anti-virus company's. Anyone that smart would look at anti-virus as unnecessary and for stupid people.
     
  13. CloneRanger

    CloneRanger Registered Member

    Joined:
    Jan 4, 2006
    Posts:
    4,833
    Even if "some" AV etc detects 99.9% even EVERY day, which i Very much doubt !

    Out of the "supposed" 80,000 new nasties released Every day, that equates to 80 missed Every day X 365 days = 29,200 per year missed. Even if it were half the 80,000 it's still = to 14,600 missed every year !

    Now calculate for less that 99.9% & you'll seen realise why Lots of people still get blasted Every day around the world, & will continue to with just AV.
     
  14. RejZoR

    RejZoR Registered Member

    Joined:
    May 31, 2004
    Posts:
    6,426
    If you say so:rolleyes:
     
Loading...
Thread Status:
Not open for further replies.