AV Testing

Discussion in 'other anti-virus software' started by CloneRanger, Mar 7, 2010.

Thread Status:
Not open for further replies.
  1. CloneRanger

    CloneRanger Registered Member

    Joined:
    Jan 4, 2006
    Posts:
    4,978
    Several curent threads on this topic, with various people suggesting the tests are done in default mode. I perfectly understand why they say this, but i feel in order to tests the full strengths of any product, they should be done with ALL settings on Maximum.

    Otherwise, as we've seen, some products "appear" to do worse than they are actually capable of.

    Nothing against default settings in tests as such, but how about then doing another with Max settings to compare. That should sort the men from the boys :D Obviously using indentical malware files/URLs.
     
  2. GES/POR

    GES/POR Registered Member

    Joined:
    Nov 26, 2006
    Posts:
    1,490
    Location:
    Armacham
    Thats what AVC is allready for mait, realworld is what counts - default= real world protection for most
     
  3. mvario

    mvario Registered Member

    Joined:
    Sep 16, 2008
    Posts:
    339
    Location:
    Haddonfield, IL
    I would say configure for the intended audience of the test. If the test results are aimed at "average" users then default settings are appropriate. On the other hand if someone was doing an informal test for a more advanced audience, say Wilders forum users, then, well, does the average Wilders forum user use default settings? If not then perhaps changing the settings would provide more useful results for that particular audience.
     
  4. SAW

    SAW Registered Member

    Joined:
    Oct 25, 2009
    Posts:
    48
    Default is what the maker considers is the best blend of protection and performance. Many AVs can be set to stop anything however if their FPs go way up and/or they use up so many resources that they become worse than or as bad as a virus themselves they are of little use to most. In order to set on the max for tests then you'd have to add in the FP rate and tests to gauge the performance hit, it would be way too hard and or time consuming for one person to do very often. Plus it would involve human decisions as to does setting x increase max detection or not and does it over value the performance hit. Default is probably the best way to test.
     
  5. CloneRanger

    CloneRanger Registered Member

    Joined:
    Jan 4, 2006
    Posts:
    4,978
    Appreciate your opinions, but i still feel that unless each AV is configured and tested to it's FULL capabilities, then they will All obviously fall short to a greater/lesser degree than they otherwise could do.

    No point in having protection, if it doesn't, and as much as possible :D
     
  6. trjam

    trjam Registered Member

    Joined:
    Aug 18, 2006
    Posts:
    9,102
    Location:
    North Carolina USA
    Why? To the 2 percent that will ever enable the strongest settings or the 98 percent that will use it out of the box. Testing for the 2 percent is not the true test. It just isnt.
     
  7. Fly

    Fly Registered Member

    Joined:
    Nov 1, 2007
    Posts:
    2,201
    How many people here are 'average users' ?
     
  8. TonyW

    TonyW Registered Member

    Joined:
    Oct 12, 2005
    Posts:
    2,741
    Location:
    UK
    I think we can exclude Wilders users as they take the time to read up on these things. It's why we're here. ;)

    Most of us should be smart enough to be able to configure the anti-malware program of choice to its fullest potential for our needs. But then again, some of us are happy with default or medium settings. It's your own experience that counts.
     
  9. Page42

    Page42 Registered Member

    Joined:
    Jun 18, 2007
    Posts:
    6,941
    Location:
    USA
    trjam is correct. Testing should absolutely be for the so-called 98%. The developer decides what settings are best for the majority of users, and that is what a product should be tested on. Testing for the 2% would be entertaining, no doubt about it... but not practical, or at least not applicable, to the broadest range of users.

    Besides, it would be misleading to test something that is tweaked to the max and then release it to the public in a lesser (default) setting.
     
  10. icr

    icr Registered Member

    Joined:
    Sep 6, 2008
    Posts:
    1,589
    Location:
    UK
    Most of the novice users never touch the confg of any AV they keep as it is.:) :)
     
  11. TonyW

    TonyW Registered Member

    Joined:
    Oct 12, 2005
    Posts:
    2,741
    Location:
    UK
    And most never come to forums such as this.
     
  12. xxJackxx

    xxJackxx Registered Member

    Joined:
    Oct 23, 2008
    Posts:
    8,625
    Location:
    USA
    But that 2% are the users of this site. Why do tests for the 98% that aren't here? I'd rather come here and read results relevant to my usage, not another set of "me too" results that I could see on a zdnet review.
     
  13. bellgamin

    bellgamin Registered Member

    Joined:
    Aug 1, 2002
    Posts:
    8,102
    Location:
    Hawaii
    Well said!
     
  14. 19monty64

    19monty64 Registered Member

    Joined:
    Apr 10, 2006
    Posts:
    1,302
    Location:
    Nunya, BZ
    Testing for the 2% gets appreciated. :) Testing for the 98% pays the bills. ;)
     
  15. Page42

    Page42 Registered Member

    Joined:
    Jun 18, 2007
    Posts:
    6,941
    Location:
    USA
    Who said they're not here? I always figured that this site was created as much for the beginners and intermediately skilled computer users as it was for the geeks among us. You may be fooled into thinking that everyone here is proficient in computer and software use, but that would probably be because a great many newbies are lurking, trying to learn all that they can, quite possibly too intimidated to join into the discussions for fear of revealing their "2%" status. So we see a preponderance of posts from the more knowledgable folks. It's my belief that the 98% are here, but not necessarily in plain sight. I could be wrong, of course. :)
     
  16. ameyap

    ameyap Registered Member

    Joined:
    Feb 16, 2010
    Posts:
    87
    i agree with trjam.. its the security vendors responsibility to determine the best possible settings for their particular softwares
     
  17. Hugger

    Hugger Registered Member

    Joined:
    Oct 27, 2007
    Posts:
    1,003
    Location:
    Hackensack, USA
    Agree!
     
  18. TonyW

    TonyW Registered Member

    Joined:
    Oct 12, 2005
    Posts:
    2,741
    Location:
    UK
    Yes, if those people actually take the trouble to investigate and "lurk" in forums like this. However, not everybody reads forums. None of my family do, for example, & they're part of the 98% that need protecting.

    It's people like these that forward bogus virus warnings; I've received some from them and I've had to reply explaining the hoax nature of such mails. I even find relevant links so they can read up on it. Some people just haven't got to the point of checking things out themselves even though it's actually easy to do that.
     
  19. CloneRanger

    CloneRanger Registered Member

    Joined:
    Jan 4, 2006
    Posts:
    4,978
    Hi gang.

    My main point is, if tests were done with higher/max settings, as well as default, and compared side by side so people could easily see the difference, then more of them could be encouraged to choose them. Especially if they were shown how to do it, i believe they might prefer to be better protected, why wouldn't they.

    And wouldn't that be better all round ?
     
  20. trjam

    trjam Registered Member

    Joined:
    Aug 18, 2006
    Posts:
    9,102
    Location:
    North Carolina USA
    I agree but, how many normal joes are even looking or care about a site like this. To most, install and forget. They are more concerned about how many quarts of oil their pickup truck is down.
     
  21. xxJackxx

    xxJackxx Registered Member

    Joined:
    Oct 23, 2008
    Posts:
    8,625
    Location:
    USA
    My thoughts exactly. Show me what these products are capable of. If there are uneducated lurkers here, let them learn something for their effort of coming here at all. :thumb:
     
  22. dawgg

    dawgg Registered Member

    Joined:
    Jun 18, 2006
    Posts:
    818
    The benefit of doing 1 is high. The benefit of doing 2 is tiny. Considering the costs of doing 2 double, its a no-brainier really as to how many should be done.

    Of course, if you are expecting volunteer testers to do it, then it's totally up-to-them how they want to test, but IMO, testing in full-blown max is not always really helpful as then some AVs at max can block almost everything from running and then always get 100% - really does not show much. And if you wish to utilise the AV to the best of it's ability, for the real power-user (inc sandbox), then its quiet pointless doing it.

    Maybe its more sensible only making scanning engine max - although the argument regarding FPs may arise as usual.

    Some comparatives (inc AVC) ask vendors which settings they wish to be tested with (and on-demand and FP tests are done with the same settings).
     
  23. skokospa

    skokospa Registered Member

    Joined:
    Apr 1, 2009
    Posts:
    177
    Location:
    Srbija
    Fully agrees
    Manufacturers antivirus software set by program to after installation meet the needs of 98% of users.
    Anyway, see the methodology and manner of testing AV Comparatives http://www.av-comparatives.org/comparativesreviews
     
    Last edited by a moderator: Mar 13, 2010
  24. YeOldeStonecat

    YeOldeStonecat Registered Member

    Joined:
    Apr 25, 2005
    Posts:
    2,345
    Location:
    Along the Shorelines somewhere in New England
    As soon as you start doing "tweaking/adjustments"...you open up a whole world of debates. What the tester might consider to be the best and strongest settings, thousands of users of that product will complain that those settings aren't correct.

    Default settings avoids this debate, keeps it a level playing field.
     
Thread Status:
Not open for further replies.
  1. This site uses cookies to help personalise content, tailor your experience and to keep you logged in if you register.
    By continuing to use this site, you are consenting to our use of cookies.