AV-Comparatives Results - Nov 2007 Retrospective/ProActive Test

I have nothing against AVIRA, Bitdefender or whatever.

 

You definitely have nothing against either surely and you do an excellent job in testing out the AVs. Only thing is your penalty process can do with a less drastic rating system.

 

the penality system should be (but is/will not be) applied as well to the test of february and august. so, avira would there get only A instead of A+. but no one of you guys think about how they would score without an aggressive heuristic. False positives are an important issue, passing our FP test should not be that difficult. i think i said 6 months ago that i plan to penalize products already when 1-3 FP's occur, but looking on how they all still produce FP's, I did not introduce that. when AVIRA will have much lower number of FP's but still be able to provide high detection rates they will be able to be proud of what they got, changing the rating system now to make them look better would not be fair and is not in the sense of the award system.

 

I use the Avira Security Suite and WinPatrolPlus on 79 computers that are always on and are in a high risk mode. We have never had any kind of virus/malware and never a false positive. There is an incredible amount of paranoia on this site; I find it almost comical at times.

 

Well, penalizying them now will probably make them reconsider their engines again.
I still don't get 9 FPs with no penalty and 16/19 with 2 penalties, but anyway it's your test... and if you consider this way, ok.

 

Well done kaspersky and nod and to others

thank you IBK for all your hard work much appreciated

i know im gonna get flammed for saying this but here it goes :

everytime i see the results from IBK and some ones av product never did well or that ww2 starts_

If antivirus a beat antivirus b it doesnt mean it better all round and stop getting in a 2n8 most off the samples im sure we would never come across anyway,

if the antivirus what ever it is works well for you and you enjoy using it keep it no matter what test site said

Now getting ready to hide under sofa

 

HUH? Avira's FP's are almost all signature not heuristic and that makes them even worse. I think IBK should penalize them further for all the signature FP's.

 

Paranoia feeds this market........

 

Screw AVs. I have gone virtual and am going to quit worrying about all of this stuff. Thanks Peter and Blue.

 

What the hell does paranoia have to do with FP's that cause all sorts of problems and exist only because the AV vendor wants to be number one in detection and doesn't care how many FP's they produce in order to do that? What's wrong here is all the folks who should know better than to claim FP's are inconsequential. I am really surprised at what some have been saying here. And those who criticize IBK and claim he is biased....geez...that is really low.

 

Never had a FP with Avira so I cannot comment. I do my own research, so I would have to take your word for it that the FP's are "almost all" signature based.

That said, are you still using Avira with all the flaws and additional penalties you think IBK should add to it?

 

It sure does.

 

Not have had one FP, 2 years with Avira, max heuristic, extended threat enabled.

Mele, maybe you should use Eset, its the darling of the test.

 

I agree with that.

 

At the moment, yes, I still have Avira installed but I have Guard disabled. I also have Avira Suite ver. 8 beta installed on two virtual machines. I haven't seen any FP's on those machines but one is a new install of XP Pro SP2 and has very little on it and the other machine is my main virtual machine with XP Pro SP1. Its virtual disk is almost full but virtual disks are small and so, yeah, it has applications and printscreen files, etc. but nothing like all the stuff on my host machine where all the FP's keep being found by Avira. So, I think someone who only uses email, and a little bit of surfing to a few sites, might never get an FP with Avira. Avira appears more and more to be aimed at the naive, beginning user and they don't seem to want any advanced users. They have made that clear anytime we advanced users have asked for improvements. It is the user who downloads a lot that is most in need of an AV and it is that very user that Avira produces the majority of FP's for. Almost all FP's, actually maybe all, that I have seen from Avira are in my downloaded programs folder which is rather gigantic because within that folder I have a file that contains all downloaded programs from my first XP computer. I have an AV mainly to scan stuff I download and that is where Avira is producing all the FP's. Granted that is much better than alerting on an essential system file. But when you have an AV mostly to determine if downloads are ok ...well...

 

I told them their firewall did not stealth all ports, and they bombarded me with a bunch of evasions and obfuscations. My personal experience is that they do not like reasoned and evidence backed criticisms (even if intended to be constructive). There is also a language barrier in my view, and they seem to not fully understand when they translate from english to german and back again. But I put up with it for excellent Avira detection-with all it's flaws.

 

This is a graph of False Positives / Proactive Detection Rate. Actually Avira did better than some AV rated Advanced.

 

You have a valid point. They must got one penalty too.

I don,t agree with this.

 

Indeed, still don't see the reason for the harsh penalty but then it doesn't really matter.

 

Folks,

Let's not lose sight of the larger picture in this discussion. As noted in both the on-demand and retrospective test reports, all products tested come from a good group of products and, if used correctly and kept updated, should keep a user safe.

The consolidated ratings use a tester developed metric to differentiate performance obtained in the testing protocol, but performance and suitability for a specific user are really different traits. If I am recommending a product without detailed context, I'll oftentimes err on the side of weighting tester provided performance metrics heavily. Yet, in my own usage, I will employ products that are not at the top of the test performance heap if they possess additional traits that I value - and that could be anything from cost, conflict with other software, performance impact, or perhaps a user interface that I stand a shot at navigating.

Like any of the test results that I may read, I use a personal filter when I weight the final outcome. We obviously all have different penalty functions for false positives (and net detection %, and so on), but that doesn't mean our view is right and everyone else's is wrong. It just means we weight certain traits differently.

Blue

 

As I posted earlier, the process involves a lot of subjectivism. That is simply the way it is. Each person making his or her rational decision based on criteria that are important to you. If is fallacious to argue about the meaning of objective (hopefully) results of tests, when AVs will be employed by different people for different reasons. I currently use Avira and have a license for NOD32 as well because their detection rate is what I value most.

 

Exactly.

Although there are hard numbers in the background, the final certification levels on both the on-demand and retrospective tests involve subjective decisions. The best one can do is to have internally self-consistent approaches.

Taking Avira as a specific case, while the number of false positives is not a lot larger than, for example, Avast, it does appear to be from a distinct population, as indicated here. Up to this point the analysis is quite objective - these are the numbers and here are some broad categories. There is then a large subjectively based leap regarding how that categorization should play out in determining a final ranking.

That's simply the nature of the beast.

Blue

 

Agreed. A good heuristic is one which provides great detection with a low amount of FPs. And heuristic detections do sometimes affect the on-demand comparative scores by a significant margin.....

 

Hi Blue:

FWIW, I agree 100% with you here.

AV produces their reports and some users look at these results and make their judgments. Some posters don't seem to have read the whole report and all the caveats about not relying on the %'s.

If I use product X and see it is top of the heap in this report
I am pleased it is only second nature to have our previous choice of AV "validated". We don't seem to challenge IBK's methods in this case do we?

If on the other hand my earlier choice fell off the top of the heap well then WW3 posts start up the numbers are wrong , the process unfair, FP are too heavily weighted etc etc.

I am free as are all posters to create my own method of selecting and use that to "adjust" the results. Thing is we need to keep our selection method the same and not adjust it just to show once again that my first choice was right all along! That is intelectual dishonesty with yourself

 

Just a general personal comment here, when one reads the result of a test like this, or any other evaluation/test result, one needs to ask "What does this teach me?" That's true of any discussion and why A vs. B product threads tended to be so pointless - they morphed into product advocate/basher flamefests that had no chance of teaching anyone anything.

When I examine these tests, particularly in this case since a historical trail is available, I try to look at trend data in addition to the raw results. When I do that, what I learn is:

• Proactive detection has only nudged a bit since May 2007. I realize that the tests are slightly different (1 month vs 3 month equilibration phase), but the result average went from 38.4% (May 2007) to 39.4% (Nov 2007) - Fortinet results excluded. There hasn't been a huge leap in performance over the entire course of the proactive test series with the average spanning a net range of 27.1 to 39.4%. Yes, some products are starting to excel in this category, but as a whole the industry has not developed a proactive detection style solution. Note - in making that statement I am excluding alternate approaches such as HIPS modules, etc, now in some products.
• In terms of current result, we have Avira/NOD32, then Trustport, the main field (AVK, Avast, BitDefender, DrWeb, Kaspersky), and then trailers. The "trailers" are fine products in their own right and are really a part of the main pack of commercial products
• Avira is continuing on a road of deliberate improvement started about 2 years ago. They have steadily increased proactive detection levels, and while false positives do remain somewhat high, they have not gone through the roof as valid detections have increased. While NOD32 is generally acknowledged to be the industry leader here, Avira has run alongside NOD32 over the past couple of years with the primary difference now resident in false positive count. Overall, this is an excellent result.
• Avast continues to be impressive, also showing steady improvement over the past couple of years. For a free product, IMHO, this is the one to use and the paid option is a top contender as well
• While KAV has improved relative to the two prior proactive test results, it's really rather similar to AVK/BitDefender/DrWeb and TrustPort for that matter in that performance has been steady, albeit somewhat static over the longer view. It's back to where is was, which is very solid, but has not taken it to the next level. Certainly, that result is in part due to the focus on the Proactive Defense module, which is arguably a more robust global solution.
• F-Prot has certainly been pulling itself up over the past year or so and is now in the main mix. That wasn't the case prior to the currently released product. Again, an excellent result.

So, what's it all mean to me?

I still think having an AV is important since it provides indirect access to knowledge about malware that really only exists in the hands of the experts. However, it probably needs augmentation.

HIPS and other process firewalling schemes are a viable solution if that's your bag and you don't surrender before completing the learning curve. Firewalls are also useful, but I don't see them as the solution.

Limited User Accounts are a solution, as long as all your applications and usage style are conducive to this type of environment. Sandboxing works as well, although my personal preference is tending towards what is probably best generically characterized as lightweight virtualization - it's easy to implement/understand/use, appears quite robust thus far.

I'm sure there's a gap or two to be found, but lets be realistic, something like a decent AV (i.e. any of the products examined by IBK as well as others with similar behavior) used in combination with a virtualization product (Returnil, PowerShadow, ShadowDefender, ShadowUser Pro, etc.) is a potent, very light, cost effective approach. It requires some user action to enter the virtualized state if this is not constantly on, and a user initiated restart to clear things out, but the up-front demands on the user can be quite minimal, it can be implemented for a novice user quite easily, and used by them with no significant learning curve.

Blue