AV-Comparatives: Real-World Protection Test October 2014

Discussion in 'other anti-virus software' started by FleischmannTV, Nov 12, 2014.

  1. avman1995

    avman1995 Registered Member

    Joined:
    Sep 24, 2012
    Posts:
    944
    Location:
    india
    Anon,again you assumed I am defending X AV. I never mentioned that i was defending it.I am talking universally.

    Even if all vendors are vested in these figures.I still dont care.You are yet again missing the point!!

    Here in the real world we have like 50,000+ samples coming out everyday.So there is like millions coming out every month.And they are jut testing only with a few hundreds.Again on-demand test proves nothing because again sample set is limited and second is that no execution of samples.And third is that the test is carried long after the infection took place.The time gap between the actual release of a malware and it infecting a system is too small and today's malware grows fast.I think even Stefan will agree that.

    I never said I as defending any AV.I actually stated in the avast! thread that the performance is poor.But if you look at the facts again these tests are kind of useless from today's point of view.Oh and by the way,its my right to voice my opinion.

    Also if I take 50,0000 samples for 1 day and calculate 4% miss on it.Its 2000 samples missed.I dont think that's small number?? And I am sure even after AV gets all those 2000 there is going to be a significant amount of a 100's that's going to go past undetected.Again,the malware goes up and down so fast in today's world its impossible to calculate.And there is a array of malware families out there and some just go unclassified.I am sure every malware writer will able to bypass a X AV with X varient of X malware just by using social engineering tactics,Packing it with new packer,Polymorphic tactics.And if you look around a bit,there you have ransomwares,USB trojans which change every hour.

    Whereas in a 24 missed sample set which a baby in front of 2000 every AV will easily able detect them all.

    Just my opinion guys.Please don't go around with some blemish rant once again.If you have a valid explaination then maybe we could debate further on the test methodology.
     
    Last edited: Nov 14, 2014
  2. anon

    anon Registered Member

    Joined:
    Dec 27, 2012
    Posts:
    6,811
    Yes math teacher, whatever you say.
     
  3. avman1995

    avman1995 Registered Member

    Joined:
    Sep 24, 2012
    Posts:
    944
    Location:
    india
    :blink::p;)
     
  4. ance

    ance formerly: fmon

    Joined:
    May 5, 2013
    Posts:
    1,359
    Haha, well said. :D
     
  5. Noob

    Noob Registered Member

    Joined:
    Nov 6, 2009
    Posts:
    6,491
    Thanks for the update.
    Good results for Emsisoft IMO. :D
     
  6. aaa839

    aaa839 Registered Member

    Joined:
    Oct 11, 2012
    Posts:
    253
    Location:
    Hong Kong
    Put more energy APC
     
  7. redcell

    redcell Registered Member

    Joined:
    Sep 27, 2010
    Posts:
    126
    How come Comodo Internet Security is not included?
     
  8. garrett76

    garrett76 Registered Member

    Joined:
    Mar 18, 2014
    Posts:
    221
    Because there was a quarrel between Comodo and AV-Comparatives :)
     
  9. 142395

    142395 Guest

    Actually not. That can only be applied when sampling was unrestricted random sampling, and in this case obviously it's not.
    Also note, in statistics random sampling is not the only one method and not always necessary. Positive/purposive sampling is another major and valid method, and obviously AVC use this at least as a part of sampling method.

    Remember if condition is strong enough (all variables which affects result are known and only them affects result), then very small sample is enough. Of course it's not common in real situation, so some device is needed.
    AVC apparently stratifying malwares. Stratification is quite important especially in positive sampling. As an example, they classify malware into families, right? So all malware will be classified to any of families according to architectural similarity or other properties. If this classification is perfect and all prevalence data are known, just take one sample from each family and weighting the result according to perfect prevalence data is enough.
    Reality is not. There's no perfect classification, it is possible that malware a1 in family A can be detected by product X but a2 in A can't be detected because there're many factor which was not considered when AVC classified (especially reputation data can be independent from architectural similarity).
    So it is better that instead of drawing one sample from each family, draw samples from a family according to prevalence of the family. If family B is twice as prevalent as C, then draw twice samples from B compared to C. And also when draw samples from each family, draw it randomly. This can neutralize other missed factor which can affect the result. This is an simplified example of stratified random sampling with proportionate allocation.

    This method minimize the random error by stratification (when sample size is small, random sampling can cause large random error) while also minimize sampling bias by random sampling.

    I don't know what method AVC use and strongly hope they clarify much more (their PDF is far from detailed explanation, but sadly it is the best among those organization), but that method is very likely (of course there're other methods).
     
  10. 142395

    142395 Guest

    No offence intended but you're ignoring statistics.
    If it was unrestricted random sampling, I would agree with you. But it's not.
    Those handful of samples are representatives, not a randomly gathered malware.
    As long as AVC's classification is proper, it makes enough sense.
    Also they divide the results into some clusters. They say difference in the same cluster can be random error, but different cluster means it is unlikely that product X was mistakenly assigned wrong cluster, more precisely the possibility that X which have to be classified to cluster 1 was wrongly classified to cluster 2 is less than 5% (5% is conventionally used in statistics).

    Well, still error can happen. But the fact Avast got low score not only once minimizes such possibility. Also bias or not proper classification can be, but the fact Avast got low score not only AVC minimizes that possibility too.

    If you still continue to say same thing, it'll look like you ignore and disregard well established statistics. Then you also have to disregard many of useful facts in social science and human science, partly even in natural science, because they're based on such statistical method (gathering tons of samples is often impossible, not limited to AV testing).

    I have no intention about Avast, actually I don't care about detection rate things. In avast, just enable hardened mode will eliminate almost all of those threats.
    But I can't ignore statement which seems to ignore or disregard statistics, as a student in mathematical science.
     
  11. SweX

    SweX Registered Member

    Joined:
    Apr 21, 2007
    Posts:
    6,429
    I would ask Comodo that question :rolleyes:
     
  12. Minimalist

    Minimalist Registered Member

    Joined:
    Jan 6, 2014
    Posts:
    14,447
    Location:
    Slovenia
    @142395
    Yes I agree. I was speaking in general - assuming that we have a database of all malware and then randomly select our sample.

    Prevalence of malware is only one factor we can use to get our sample. We can also consider other factors like:
    - how prevalent is some malware family or sample in different countries or regions.
    - what kind of user are you? Gamer, P2P user...

    I remember few years ago there was a test where results were adjusted for different user types. They estimated how likely different user types will encounter different malware (families) and adjusted results according to that. This way user can choose which group they are member of, and then check which AV would best protect this group.

    It was interested research but I can't find it any more...
     
  13. malexous

    malexous Registered Member

    Joined:
    Jun 18, 2010
    Posts:
    830
    Location:
    Ireland
  14. Minimalist

    Minimalist Registered Member

    Joined:
    Jan 6, 2014
    Posts:
    14,447
    Location:
    Slovenia
  15. 142395

    142395 Guest

    I happened to write about regional difference firstly, but removed because it made explanation longer and more complicated. (and statistics is not my majoring...)

    You can use multi-stage sampling to address those multi-factor issue.
    However in general, multi-stage sampling can cause more risk of random error, so how to sample cluster is quite important.
    In this case firstly stratify regions (what have to be stratified is regions, or IOW set of all malware in each regions have to be stratified as a cluster, not as a individual sample), properly allocate them (proportionate allocation is one method, but not limited to this), then stratify each cluster according to another factor (e.g. world prevalence), allocate again and draw actual samples (randomly). If there's another known factor, you can repeat same process, but often 2-stage stratified random sampling is used. Other factor or even unknown factor are (somewhat) cared by random sampling.
    Note reader can get both of regional results and worldwide results (it also reflects regional differences), though currently AVC don't offer such a multi-results.

    That PCSL test profiling seems to be just a simple weighing, so I'm skeptical about it's usefulness.
    Also it combines FPs, performance, phishing detection etc. but I think most people want to know protection, performance, and FPs separately.
     
  16. guest

    guest Guest

    Oh wow, a bit surprising to see Panda got overthrown this month. What happened to the cloud?

    And oh mai gawd Fleischmann, I love your current avatar! Is that a destroyer? What class and from which Navy? :ouch:
     
  17. siketa

    siketa Registered Member

    Joined:
    Oct 25, 2012
    Posts:
    2,718
    Location:
    Gaia
  18. SweX

    SweX Registered Member

    Joined:
    Apr 21, 2007
    Posts:
    6,429
  1. This site uses cookies to help personalise content, tailor your experience and to keep you logged in if you register.
    By continuing to use this site, you are consenting to our use of cookies.