AV-Comparatives: Real-World Protection Test October 2014

Anon,again you assumed I am defending X AV. I never mentioned that i was defending it.I am talking universally.

Even if all vendors are vested in these figures.I still dont care.You are yet again missing the point!!

Here in the real world we have like 50,000+ samples coming out everyday.So there is like millions coming out every month.And they are jut testing only with a few hundreds.Again on-demand test proves nothing because again sample set is limited and second is that no execution of samples.And third is that the test is carried long after the infection took place.The time gap between the actual release of a malware and it infecting a system is too small and today's malware grows fast.I think even Stefan will agree that.

I never said I as defending any AV.I actually stated in the avast! thread that the performance is poor.But if you look at the facts again these tests are kind of useless from today's point of view.Oh and by the way,its my right to voice my opinion.

Also if I take 50,0000 samples for 1 day and calculate 4% miss on it.Its 2000 samples missed.I dont think that's small number?? And I am sure even after AV gets all those 2000 there is going to be a significant amount of a 100's that's going to go past undetected.Again,the malware goes up and down so fast in today's world its impossible to calculate.And there is a array of malware families out there and some just go unclassified.I am sure every malware writer will able to bypass a X AV with X varient of X malware just by using social engineering tactics,Packing it with new packer,Polymorphic tactics.And if you look around a bit,there you have ransomwares,USB trojans which change every hour.

Whereas in a 24 missed sample set which a baby in front of 2000 every AV will easily able detect them all.

Just my opinion guys.Please don't go around with some blemish rant once again.If you have a valid explaination then maybe we could debate further on the test methodology.

 

Yes math teacher, whatever you say.

 

Haha, well said.

 

Thanks for the update.
Good results for Emsisoft IMO.

 

Put more energy APC

 

How come Comodo Internet Security is not included?

 

Because there was a quarrel between Comodo and AV-Comparatives

 

Actually not. That can only be applied when sampling was unrestricted random sampling, and in this case obviously it's not.
Also note, in statistics random sampling is not the only one method and not always necessary. Positive/purposive sampling is another major and valid method, and obviously AVC use this at least as a part of sampling method.

Remember if condition is strong enough (all variables which affects result are known and only them affects result), then very small sample is enough. Of course it's not common in real situation, so some device is needed.
AVC apparently stratifying malwares. Stratification is quite important especially in positive sampling. As an example, they classify malware into families, right? So all malware will be classified to any of families according to architectural similarity or other properties. If this classification is perfect and all prevalence data are known, just take one sample from each family and weighting the result according to perfect prevalence data is enough.
Reality is not. There's no perfect classification, it is possible that malware a1 in family A can be detected by product X but a2 in A can't be detected because there're many factor which was not considered when AVC classified (especially reputation data can be independent from architectural similarity).
So it is better that instead of drawing one sample from each family, draw samples from a family according to prevalence of the family. If family B is twice as prevalent as C, then draw twice samples from B compared to C. And also when draw samples from each family, draw it randomly. This can neutralize other missed factor which can affect the result. This is an simplified example of stratified random sampling with proportionate allocation.

This method minimize the random error by stratification (when sample size is small, random sampling can cause large random error) while also minimize sampling bias by random sampling.

I don't know what method AVC use and strongly hope they clarify much more (their PDF is far from detailed explanation, but sadly it is the best among those organization), but that method is very likely (of course there're other methods).

 

No offence intended but you're ignoring statistics.
If it was unrestricted random sampling, I would agree with you. But it's not.
Those handful of samples are representatives, not a randomly gathered malware.
As long as AVC's classification is proper, it makes enough sense.
Also they divide the results into some clusters. They say difference in the same cluster can be random error, but different cluster means it is unlikely that product X was mistakenly assigned wrong cluster, more precisely the possibility that X which have to be classified to cluster 1 was wrongly classified to cluster 2 is less than 5% (5% is conventionally used in statistics).

Well, still error can happen. But the fact Avast got low score not only once minimizes such possibility. Also bias or not proper classification can be, but the fact Avast got low score not only AVC minimizes that possibility too.

If you still continue to say same thing, it'll look like you ignore and disregard well established statistics. Then you also have to disregard many of useful facts in social science and human science, partly even in natural science, because they're based on such statistical method (gathering tons of samples is often impossible, not limited to AV testing).

I have no intention about Avast, actually I don't care about detection rate things. In avast, just enable hardened mode will eliminate almost all of those threats.
But I can't ignore statement which seems to ignore or disregard statistics, as a student in mathematical science.

 

I would ask Comodo that question

 

@142395
Yes I agree. I was speaking in general - assuming that we have a database of all malware and then randomly select our sample.

Prevalence of malware is only one factor we can use to get our sample. We can also consider other factors like:
- how prevalent is some malware family or sample in different countries or regions.
- what kind of user are you? Gamer, P2P user...

I remember few years ago there was a test where results were adjusted for different user types. They estimated how likely different user types will encounter different malware (families) and adjusted results according to that. This way user can choose which group they are member of, and then check which AV would best protect this group.

It was interested research but I can't find it any more...

 

Was this it? https://www.wilderssecurity.com/threads/pcsl-security-solution-review-on-windows-8-platform.344813/

 

Yes, THANK YOU! That's what I've been looking for.

 

I happened to write about regional difference firstly, but removed because it made explanation longer and more complicated. (and statistics is not my majoring...)

You can use multi-stage sampling to address those multi-factor issue.
However in general, multi-stage sampling can cause more risk of random error, so how to sample cluster is quite important.
In this case firstly stratify regions (what have to be stratified is regions, or IOW set of all malware in each regions have to be stratified as a cluster, not as a individual sample), properly allocate them (proportionate allocation is one method, but not limited to this), then stratify each cluster according to another factor (e.g. world prevalence), allocate again and draw actual samples (randomly). If there's another known factor, you can repeat same process, but often 2-stage stratified random sampling is used. Other factor or even unknown factor are (somewhat) cared by random sampling.
Note reader can get both of regional results and worldwide results (it also reflects regional differences), though currently AVC don't offer such a multi-results.

That PCSL test profiling seems to be just a simple weighing, so I'm skeptical about it's usefulness.
Also it combines FPs, performance, phishing detection etc. but I think most people want to know protection, performance, and FPs separately.

 

Oh wow, a bit surprising to see Panda got overthrown this month. What happened to the cloud?

And oh mai gawd Fleischmann, I love your current avatar! Is that a destroyer? What class and from which Navy?

 

https://forums.comodo.com/melihs-co...ip-and-financial-dealscontinued-t78934.0.html

 

Yes I know about that.