The danger of AV testing sites

Discussion in 'other anti-virus software' started by Bodhitree, Dec 20, 2012.

Thread Status:
Not open for further replies.
  1. Nebulus

    Nebulus Registered Member

    Joined:
    Jan 20, 2007
    Posts:
    1,635
    Location:
    European Union
    Re: The danger of sites like AV Comparatives.

    This kind of test sites is the closest thing you can get to real life behavior of antivirus software. I mean, how are you going to test your AV? Infect your computer with all kinds of malware and see if your AV is reacting?

    Of course 100% security will never be achieved, no matter what AV are you using, but checking several test site will help you evaluate if you will get a serious protection, some protection or no protection from your chosen AV software.
     
  2. Mman79

    Mman79 Registered Member

    Joined:
    Sep 19, 2012
    Posts:
    2,016
    Location:
    North America
    Re: The danger of sites like AV Comparatives.

    People here do the same thing, and this is where a lot of "experts" hang out. How many posts here consist of "Such and such only got 95%, terrible :thumbd:"? I don't however see that as "dangerous". In fact, people should base their decision first and foremost on detection rates. That's the whole point of the software. After that, does it run well on my system and not just on "everyone else's"? If it does, great, if not, move on to the next AV that did well on the test.
     
  3. PJC

    PJC Very Frequent Poster

    Joined:
    Feb 17, 2010
    Posts:
    2,959
    Location:
    Internet
    Re: The danger of sites like AV Comparatives.

    IF AV-Comparatives and AV-Test had Not Tested all AV products under the Same Condition,
    their Tests would have been seriously BIASED/FLAWED!

    IF Microsoft does Not agree with the Methodology of AV-Comparatives and/or AV-Test,
    Microsoft can easily withdraw from both Tests!

    Since you constantly Criticize the Methodology of AV-Comparatives and AV-Test,
    why don't you offer your "Improvement Instructions" to Andreas Clementi and Andreas Marx?
     
    Last edited: Dec 22, 2012
  4. silverfox99

    silverfox99 Registered Member

    Joined:
    Jul 14, 2006
    Posts:
    204
    Re: The danger of sites like AV Comparatives.

    :thumb: :thumb: :thumb:
     
  5. Mman79

    Mman79 Registered Member

    Joined:
    Sep 19, 2012
    Posts:
    2,016
    Location:
    North America
    Re: The danger of sites like AV Comparatives.

    I agree, I'm not sure how people want to test, unless they want to set up their own lab and basically do the same thing, because that's about as "realistic" as any lab is going to get. What are these "many, many other factors"? Look, you either get infected or you don't, and your machine either handles the product smoothly or it doesn't. There aren't that many factors to consider here outside of individual bells and whistles and FPs..which are covered under the tests. What are we wanting, 300 people sitting in a row of desks with each system secured differently, one doing P2P, one surfing porn sites, and so on? It's ridiculous and no testing center is going to do it. So, either we rely on the way they do it and then see how it goes on our own system, or we let the Youtube "home labs" influence us (hint: Most are idiots and pretty much do the same damn thing the pro testers do, but with less accurate results.).

    Microsoft though has little room to complain about anything seeing as how poorly they do usually, to the point where their certification was recently stripped away. Again though, detection and usability are the only deciding factors for users. What very little remains is given zero craps about by anyone other than the developers themselves or the security field. It's simply unimportant.
     
  6. PJC

    PJC Very Frequent Poster

    Joined:
    Feb 17, 2010
    Posts:
    2,959
    Location:
    Internet
    Re: The danger of sites like AV Comparatives.

    Well-written! :thumb:
    Amen! :thumb:
     
  7. Noob

    Noob Registered Member

    Joined:
    Nov 6, 2009
    Posts:
    6,491
    Re: The danger of sites like AV Comparatives.

    I still consider these tests a good way of measuring an AV detection effectiveness. While it is only a small part of the equation it still matters. :D
     
  8. iHz

    iHz Registered Member

    Joined:
    Aug 20, 2006
    Posts:
    54
    Re: The danger of sites like AV Comparatives.

    Like you say, HTTP security is very important, prevention is better than cure, on your own pc.
    But what if someone else doesn't have good security, unknowingly downloads virus, puts on their usb, and plugs it onto your pc? Then you will need a good file scanner, right?
     
  9. Mman79

    Mman79 Registered Member

    Joined:
    Sep 19, 2012
    Posts:
    2,016
    Location:
    North America
    Re: The danger of sites like AV Comparatives.

    Yes, I would. But before that USB drive ever even approached the port of my system, I'd ask just what was on that drive, were the files contained on it scanned for viruses, and so on. Then, when it does get plugged in, it'll sit there and do nothing and like it because A. Auto-play and anything like it is disabled on my system. B. My own security will have a look along with myself before any file on there gets touched.
     
  10. safeguy

    safeguy Registered Member

    Joined:
    Jun 14, 2010
    Posts:
    1,797
    Re: The danger of sites like AV Comparatives.

    Well-said. I pretty much agree with this. If I were to add something to this, I'd say that much of the responsibility comes from the marketing department for AV vendors themselves. There's nothing wrong in choosing and displaying test results which gives the AV vendor a favorable position compared to competitors. It's a common sales strategy and I perfectly understand that. What I find annoying though is how some takes things too far and comes off as borderline misleading.
     
  11. Bodhitree

    Bodhitree Registered Member

    Joined:
    Dec 5, 2012
    Posts:
    567
    Re: The danger of sites like AV Comparatives.

    Question, and reply to me privately if necessary.

    Are there any sites that correlate results from places such as Virscan.org or Virustotal to formulate an overall detection rate per product? I have been interested in finding a site/sites that are doing this, as it makes quite a lot of sense. Virscan.org for example posts constant results, to harvest these would paint an interesting picture.

    Anyone know of sites doing this? I cannot seem to find any.
     
  12. BlueZannetti

    BlueZannetti Registered Member

    Joined:
    Oct 19, 2003
    Posts:
    6,590
    Re: The danger of sites like AV Comparatives.

    I tend to disagree with the premise of this thread - sites such as AV-Comparatives and others shed partial light on a complex and fluid situation.

    As such, the results are somewhat dated once presented and other factors should influence, and perhaps dominate, a purchase decision. However, this is no different than any purchase and many of the same metrics that one applies to any purchase (i.e. you're purchasing, or should be purchasing, a product as it currently exists, not as it might exist a year down the road; organizations have a finite lifecycle, recognize that reality; performance fluctuates; and so on) should apply in this case.

    That said, an interesting site is ShadowServer and the AV Stats that they develop. There's a lot going on beneath the numbers, and if you don't take some time to appreciate the details (and the profound limitations of those and any other results), you'll miss the message behind the numbers.
     
  13. JerryM

    JerryM Registered Member

    Joined:
    Aug 31, 2003
    Posts:
    4,306
    Re: The danger of sites like AV Comparatives.

    I think that AV-C is the single most respected testing lab. I know it is for me. I am persuaded that there are those who would not be satisfied with their own labs, if they had them.
    I also disagree with the premise of the thread. If absolute perfection is demanded then nothing will be satisfactory in this world.

    "You couldn't please some people if you hung 'em with a new rope."

    Jerry
     
  14. Arin

    Arin Registered Member

    Joined:
    May 1, 2004
    Posts:
    997
    Location:
    India
    Re: The danger of sites like AV Comparatives.

    Why do people think that synthetic tests performed by AVC or AVT, use only ITZ malwares where as the detections seen on Virustotal or Virscan are essentially for ITW malwares?

    The best real world test is what you should do within the AV's trial period.
     
  15. JerryM

    JerryM Registered Member

    Joined:
    Aug 31, 2003
    Posts:
    4,306
    Re: The danger of sites like AV Comparatives.

    If I do not get infected during that period do I assume it is a 100% AV?

    How many times have I read "test it yourself" on this forum? I must say that is baloney in the case of some of us. I am not even sure anyone here has both the knowledge and facilities to do his own test. If I had the facilities I would not have the knowledge to do my own test.

    The trial period only indicates what your AV did during that period, and is not necessarily a proof as to what will happen next month. How many threats did you experience during the trial period.

    Labs like AV-C do a good job of determining the protection of the various AVs, and better than any individual can do.

    Jerry
     
  16. Mman79

    Mman79 Registered Member

    Joined:
    Sep 19, 2012
    Posts:
    2,016
    Location:
    North America
    Re: The danger of sites like AV Comparatives.

    Both methods have their flaws. Lab testing simply shows what an AV is capable of under controlled circumstances. See them for what they are, and use them to get an idea of what you're looking at should you decide to try one of the tested vendors. However, controlled circumstances aren't going to mean a thing once you're out and about on the net. That's where "real world" testing comes in. Surf the net like you normally would during the trial. Don't "push" the AV by going out of your way to try and infect yourself. If after the trial period you're clean as a whistle, chances are you can rely on the product. If you get infected once, what infected you? How serious was the malware? Was it ransomware? Was it "rogue software"? Not many are very good at catching those, so take that into consideration.

    Should you rely on the "real world" tests of others to make your judgement? HELL no. If one person runs torrent software, another likes porn, another likes to stream media from "non-official" sources and so on, those "real world" results would be all over the map..which is where "real world" testing is flawed too.

    You don't need knowledge or facilities, you need to just use the thing for a month or two, and have an extra scanner on the side to better judge how well the AV in question is doing (MBAM is perfect for this). If you don't push your AV choice by going out and looking for trouble, you have a pretty good idea of how well it will do after the trial. I don't buy the argument that what happens this month may be different from next. What, do you really think December will go off without a hitch and, suddenly, January rolls around and you're being bombarded with trojans, rootkits and fakes? What on earth are you planning to do or think will happen? If your surfing habits don't change, rarely will your risk of infection change. The topic is being made more complicated than it really is.
     
  17. Arin

    Arin Registered Member

    Joined:
    May 1, 2004
    Posts:
    997
    Location:
    India
    Re: The danger of sites like AV Comparatives.

    The answer to your obvious strawman question is obviously no.

    I agree. All I wanted to say is that the end users should try out the AVs to see how it suits his/her requirements in the end. By no means I am thrashing professional tests nor I am calling them dangerous.

    It is just an observation from my side that most people tend to reject the tests done by professional bodies as synthetic tests. Like somehow the threats used in the tests are never going to attack a normal end user.
     
  18. Solarlynx

    Solarlynx Registered Member

    Joined:
    Jun 25, 2011
    Posts:
    2,015
    Re: The danger of sites like AV Comparatives.

    Every time I see the name of the thread I feel some disagreement with it. Dangerous av tests sites?

    I understand the danger of phishing or warez or porno sites or even dubious ad sites. But these...? Then any private opinion about avs is dangerous. Yeah, they may be somehow biased, lucrative, misleading and even ridiculous as almost everything in the world.

    Are there any victims of this danger? Someone bought the av which let him down? Then other av wouldn't?
     
  19. PJC

    PJC Very Frequent Poster

    Joined:
    Feb 17, 2010
    Posts:
    2,959
    Location:
    Internet
    Re: The danger of sites like AV Comparatives.

    Well-written! :thumb:
    100% Agree with you! :thumb:
     
  20. Solarlynx

    Solarlynx Registered Member

    Joined:
    Jun 25, 2011
    Posts:
    2,015
    Re: The danger of sites like AV Comparatives.

    :thumb:
     
  21. Bodhitree

    Bodhitree Registered Member

    Joined:
    Dec 5, 2012
    Posts:
    567
    Re: The danger of sites like AV Comparatives.

    Sure you do.. One nice thing is to image your PC, then install a product and TRY to infect yourself over a set period of time.. Visit every questionable site possible, download all of the adobe updaters and porn video players you can imagine, and then see. I would place much more stock in that kind of test then I would a synthetic AV test. Simply because this kind of real test illustrates the kind of stuff YOU would encounter.

    What you will find is, some products are quite easy to infect, other products are exceptionally difficult, even if you TRY to infect. Then you can decide what to run, do you want a product that scores 99% at a synthetic test, but is woefully easy to infect a system it is on. Or a product that tests 97% on a synthetic, and is incredibly hard to infect a machine it is on? You decide.
     
  22. Mman79

    Mman79 Registered Member

    Joined:
    Sep 19, 2012
    Posts:
    2,016
    Location:
    North America
    Re: The danger of sites like AV Comparatives.

    I'm sorry, but that's an incredibly crazy thing to tell a normal user to do, image or not. As I've stated before, the best way to handle a new product is to use your system like you normally would. What you're suggesting is a test almost replicating a targeted attack, in which every consumer product out there is going to fail if you're raining bombs down on it. That's not even real world. Real world is your realistic use of your system.
     
  23. Bodhitree

    Bodhitree Registered Member

    Joined:
    Dec 5, 2012
    Posts:
    567
    Re: The danger of sites like AV Comparatives.

    As long as someone has snapshots or images, it's pretty common sense to test a product this way since it will mirror operations on your network. While not authorative, it at least provides evidence of how well a product may do on your setup. For me, I took logs of infections from other machines I would be installing the product on, then tried to infect using those methods, similar ones, and new ones from similar locations. This allowed me to determine if the common locations where these infections come from - would be protected. RT Threat audit they are called. They aren't really complex, anyone can do them, provided they have a VM, image, or snapshot to roll back to. The key is to determine resistance from XYZ product, but yes some products would be fully capable of defending from such an assault - especially products with good HIPS and HTTPS inspectors, not many, but a few do. But that's really not the only thing considered, products that do 'OK' but excel in other areas aren't discounted.

    When a new product comes out that interests me, I install it them try to 'kill' the system, then restore it back. Sometimes it takes minutes to kill the box depending on product, sometimes hours, sometimes days, and in a few cases never. To me this method is VASTLY superior to synthetic testing with honeypot threats of which the vast majority of people would never run into.
     
  24. Mman79

    Mman79 Registered Member

    Joined:
    Sep 19, 2012
    Posts:
    2,016
    Location:
    North America
    Re: The danger of sites like AV Comparatives.

    But, most users are never going to run into the threats that will inevitably be found through your testing method of basically the digital version of "Hell Week" either. That's why when I tell others about this particular website, to not get freaked out after a day here, unless they're willing to become paranoid. Some of the worst threats that get discussed here won't ever be seen by an everyday user, and many of the security setups are overkill to say the least.

    I just don't believe in using the "total war" test for anyone but geeks and actual experts, and I certainly don't believe in recommending HIPS-like products or the testing of them unless you belong to either of those two groups of people. And, quite honestly, if a test like you suggest mirrors normal operations for people, those people have serious need of re-learning about basic security, or, in the case of people who just don't care whatsoever, frankly deserve to be infected.
     
  25. Bodhitree

    Bodhitree Registered Member

    Joined:
    Dec 5, 2012
    Posts:
    567
    Re: The danger of sites like AV Comparatives.

    Yeah but it can provide useful information. For example I know what to install on my father inlaws PC because I know all of the porn sites he visits, and what they try to deploy, and what blocks them. A LOT of products don't block those I have found, therefore this test is effective not just for me, but for him.

    A good security analyst would plug in websites logged from client PC's and find a solution that works best for the threats they generally encounter. I agree this isn't for the 'average' dude, but this forum ain't full of average dudes. Most people aren't paranoid about security, with good reason, most people aren't infected every few days by a ITW. This is security theater here.
     
Thread Status:
Not open for further replies.
  1. This site uses cookies to help personalise content, tailor your experience and to keep you logged in if you register.
    By continuing to use this site, you are consenting to our use of cookies.