AV-Comparatives Results - Nov 2007 Retrospective/ProActive Test

the percentage needed for the levels is known since 2004.

 

I definitely agree with you on this, but in that case, no security app is safe from that aspect. Firewalls will give you pop ups. A novice is more prone to be vulnerable so in lieu of that, wouldn't it be advisable to put a security app with better detection.

 

Regardless of what I think of avira I don't think it should be lowered, the figures for all the fps is certainly not alot, that's what I've been saying

however there has to be some limits, and ibk has set them

I think many products will be happy with the results of this test, regardless of ratings

 

Right Now I have 126745 files in my computer (only 30GB).

 

how many of them are pictures and videoclips?
number of files or size is meaningless.

 

stefan I think its 10 or above for standard, anything lower equates to the percentage in detection for is rating.

What I would like to know, and to be published in the results is the actual percentage of fps instead of 'many' / 'few' / 'very few'

I think if the percentages were shown, people would get a more 'overall picture' of the fp problem

 

Your statement is certainly correct for most users, but some of us more experienced users don't have a problem with a reasonable number of fp's.

 

The only test that are off any value is real world test by users of any security software.

 

And thats the key. Experienced users constitute a very small minority of people using such products - the vast majority does not know much, does not know to distinguish between an FP and a true detection, and would rather care about the FP count much more than experienced users would (some corporates too, but not most I think).

AV-comparatives' results are not restricted or limited to only the "experienced class"; the aim of the tests is to provide transparent guidelines and detection numbers for people as well as vendors to see so that vendors can improve their product and people can make a more informed purchase decision after checking other factors in addition to the detection numbers. And believe it or not, a large number of people will appreciate the rating system of AV-comparatives, and indeed FPs are not welcome for such users.

Though I feel some "tweaking" may be appreciated, I find nothing really wrong with the rating system or the test methodology this time around. I would like to say kudos to IBK for continual revision and improvement of the methodology, as well as reporting the full detections and detailed information about the various FPs for multi-engine products and Dr.Web (which had some minor detection name details truncated last time).

Its not a matter of making any product look better or worse than it really is, in the end it is about the relative performance of the product. An AV with a very high proactive protection but with lots of FPs does not pose well for the average joe, and hence it is not a good purchase option for those users. And these average joes constitute at least a statistically significant portion of any vendor's costumer base.

 

"Relative" is the key word and brings in all kinds of subjectivity. Avira has clearly decided they want a high detection rate and will tolerate a number of fp's. If the marketplace (your average joe) cannot deal with that, ultimately it will affect sales and profits. NOD32 has fewer fp's and a lower detection rate. That is what they offer their users. Ultimately, the market decides what works for them.

 

Thats right, now we just have to see what works and what doesn't....

 

Just a couple of comments here....

• False positives are a very context sensitive topic, the context being how an alert is handled. The fact is, a single false positive can range anywhere from a non-event (say a product which simply alerts a user) to a system crippling event (default deletion of any flagged file).
• It's clearly a judgement call on where various lines are drawn, although some very simple visualization tools can help assess whether the categorization is reasonable. If you have a multimodal distribution of data, rank order it, calculate the average and standard deviation of cumulative subsets (i.e. start of set to point X, X increasing), and finally plot the subset standard deviation against the subset average, you'll obtain a stepped profile such as shown on the right hand side of the figure below. The data in that plot was drawn from a simulation using four delta functions with centroid values of 1, 7, 25, and 50, although the results don't really change if the discrete values are altered (say replace all 25's with 14's). Each subset is readily visualized, as are transitions between sets. If the delta functions are replaced with normal continuous distributions, the profile will be somewhat smeared but the same trends will be apparent. The mathematics behind this simple plot are transparently obvious. If you do the same calculation with the actual false positive results, you'll find that the category lines drawn by IBK (none or very few = 0-1; few = 5-9; many = 16 and above) are actually very objective break points in the data. How that categorization impacts the final ratings is certainly open to debate, but the categorization presented is quite sound to within the scope of the data.
• How the categorization should impact the ratings is a separate judgement call, and as the discussion has indicated, rather sensitive to the user. As just noted, it is also very sensitive to the default action taken by a product on identification of potential malware since that is how most products tend to be employed. Obviously, a default action of file deletion or quarantine without user input is much dicier than a simple alert to the user, even an inexperienced one.

Blue

 

False positive can cause more trouble than a virus . Hope you guys remember what Norton did in china . I like IBK's methodology , but the penalty seems to be too much . Avira should get at least ADVANCED for 81% detection and only 16 fps. well thats just me ......

 

i agree with 031...

 

Avira should be higher, but thats ok. I will take the better detection. I like Eset but it is fairly well know their detection is not as good. But no FPs, of course you could argue the more detected, the higher chance of FPs. So I see the score 1-1 for both, but not reflected in their overall rating. It doesnt matter. Choose what you want, the majority are all good and that is about all we are going to accomplish here today.

 

exactly i myself would rather have higher detection rates since i would know for them most part if its a regular file being flagged but the average user like say my mom would not. i have nod on her computer for that very reason so she should not have to worry about stuff like that.. i dont care to much about a few fp's. now just to find some kind of discount on avira which i doubt will happen lol..just a bit to high for me..

 

thought I was finished. There are no winners or losers here. Just better educated consumers and vendors. So thank you IBK for all. Guess we will all argue again in 3 months.

 

At the end of the day, you have the statistics and can judge for yourselves with you consider acceptable (detection or FPs). Of course, everyone has different perspectives

 

great post

 

I always find av-comparatives to be very interesting and useful.



Thanks IBK!

 

Regarding the FP debate, a lot depends on the kind of user you are. If you do P2P and go to a lot of dangerous category websites, AntiVir would be best. If not, having an AV with a just good detection rate but with fewer FPs is better.

 

I could not agree more.

 

Only ESET and McAfee detected unknown macro viruses. I'm a bit dissapointed by F-Prot and Avira, both of which have known experts in macro viruses.

Detection of otherOS malware and other malware continues to be poor

I find some weirdness in McAfee's FPs
New Malware.dq variant is reported as signature FP but, New Malware.bj variant is reported as heuristic FP?

F-Prot FPs are confusing as well (no name detections).

I do agree with Stefan that Kaspersky has higher amounts of heuristic FPs.

If IBK makes a retrospective test including PUPs (Vundos, rogue apps) all the AV would score much lower.

ESET isn't the king of speed anymore They got crushed by Symantec and F-Prot. What happened to assembly programming and the such? Fortinet's speed is close to the average read speed of a moderm HDD although they have a very basic scanner. Which AVs are optimizing their engines for multi-core operation?

ESET: 71 % and 0 FPs Congratulations again

AVG and Avast: silently improving Wonder how Avast 5 will perform (emulation-based heuristics)

Kaspersky detected eScan and F-Secure packages. Isn't that funny? LOL

 

Yep, indeed that's what IBK explains on his test's report

 

A) I get the feeling that some posters think that an FP is much much worse than getting an actual infection. I would think that an infection is equally as bad (or worse) than an FP.

B) Consider -- there were 25036 malwares in the sample. I look at it this way: a missed detection equates to an INFECTION, right? Therefore, in 25036 attacks by malware...

Avira allowed me to be infected 4658 times.

KAV let me get infected 14911 times.

McAfee let me get infected 16548 times - holy guacamole, Batman!!!

C) Now then, please give thought to the effect of an FP when compared against an actual infection...

1- When I get infected, I'm infected. PERIOD! That isn't a false alarm. That isn't a minor inconvenience. I'm bloody well INFECTED. My only hope is a back-up copy, or an app that fully cleans the infection without screwing-up anything else in the process.

2- When I get a FP, I am NOT infected.
a- If I have half-a-brain then I at least do a back-up before deleting the wrongly-identified-file (WIF).
b- If I have 3/4 of a functional brain, then I do a bit of research BEFORE deleting the WIF.
c- If I am brain-dead & take zero precautions before deleting the WIF, then the fact that my computer shortly thereafter does a cyber-fart (or dies) SHOULD alert me to the fact that I have deleted an essential file.

d- Notice- someone who deletes a WIF will know about it pretty doggone quick. Someone who gets infected might not realize that fact for a long long time -- & in the meantime he screws up the computers of lots of other folks who have contacts with his infected computer.

D) In summary...

1- Somewhere along the line I think people have unintentionally played-down the seriousness & inconvenience of becomeing infected. Thus, some folks over-emphasize the big deal about an FP.

2- In my example above, one of the AVs I listed would allow me to get INFECTED 16548 times. And someone calls that *Advanced*? Lawd luvva duck -- that's flaming ridiculous!

3- This is an era of rapidly expanding technology. There are more dials & switches on my VCR than there were on the Stearman bi-plane I flew in my youth. IOW, folks have gotta learn to function in this era. If they are too lazy to do so, that's their right -- well & good. Just PLEASE do not excessively dumb-down the security apps (such that 16548 infections becomes "advanced") in order to accomodate them.