AV-Comparatives June (May 2007) Results (Retrospective / Proactive Tests)

Re: AV-Comparatives June Results (Retrospective / Proactive Tests)

Agree and disagree. But not everyone that finds that test will bother to check out FP frequency, categorical detection of specific malware types, and so forth - so a general rating system is a nice thing to have.

 

Re: AV-Comparatives June Results (Retrospective / Proactive Tests)

That is true, but it does seem to contradictorary to the line I quoted before i.e.

 

Re: AV-Comparatives June Results (Retrospective / Proactive Tests)

IBK, lets say drweb gave you permission to test the beta drweb on this test, would you?

i remember you doing an fprot one, was that because IC asked you to?

----------
as for ideas and things to improve the tests, i never thought you listened to them to be honest

 

Re: AV-Comparatives June Results (Retrospective / Proactive Tests)

NOTE: In the following discussion I use Avira & Avast for purposes of illustration only.

In this report I note that Avira detected 71%, & Avast detected 26%. Contrary to the respective detection rates, Avast is rated "Advanced" whereas Avira is rated "standard." Per IBK's comment, I assume the difference lies in the relative number of FP's.

I recognize that an AV which consistently cries "Wolf (i.e., manifests high FP's) is likely to produce user frustration, &/or lackadaisical attitude toward a real threat. On the other hand, if I got infected by using (e.g.) Avast, I would take little comfort from someone telling me, "Yes, but at least you weren't subjected to a bunch of False Positives."

In the report's UNidentified ratio of trade-off between (1) detection rate <VERSUS> (2) FP rate, the question becomes: How much reduction in attribute #1 is a "typical user" willing to accept in order to obtain a reduction in attribute #2? More to the point, who will define "typical user?"

Concerning this matter, I note that Avira's scan heuristic can be set at "Low" "Medium" or "High." As a totally unsupportable wild guess, I THINK it quite likely that Avira (at scan setting of Med or Low heuristics) would have attained better ratios of {Detection % TO FP %} than was the case for those AVs rated Advanced or higher.

IMO, Avira's scan-sensitivity-configurability means that it is tantamount to THREE different scanners, whereas AV-Comp evidently tested only ONE. The same situation may apply for other tested AVs, as well.

BOTTOM LINE- In my rash & totally UNexpert doofus opinion...

*AV's with configurable scan sensitivities should be tested separately at each setting

**There should be a quantified expression of the RATIO between detection rate AND FP rate.

 

Re: AV-Comparatives June Results (Retrospective / Proactive Tests)

@IBK: Sorry, I'm probably missing something but isn't there something wrong with F-secure false positive count ?

In the report, there is only 1 fp mentionned (both in the summary and in the detailed table), whereas the table and the conclusion say that it triggers many false alarms (hence its "standard" award instead of "advanced").

 

Re: AV-Comparatives June Results (Retrospective / Proactive Tests)

Hmm....Considering F-Secure uses Norman's engine also I think F-Secure did have a lot of FPs. Probably the contradictions in the report are a mistake, I think it should be corrected soon. Because it seems that out of habit, F-Secure's stats were put in some sections the same as Kaspersky (because until now it has always been F-Secure, eScan, Kaspersky --> same results, but this changed with 2007 version). So I think this is a simple oversight.

 

Re: AV-Comparatives June Results (Retrospective / Proactive Tests)

i cant wait to see the how the kaspersky heristics are.
they should beat the f-secure heristics.
lodore

 

Re: AV-Comparatives June Results (Retrospective / Proactive Tests)

Yes, I've been surprised not to see the Norman Sandbox FPs in F-secure's list. But there seems to be only one fp coming from the sandbox (W32/Malware, Windows XP patch package). The W32/Zacryxof.A.dropper is due to the coupling of the sandbox with the signature engine.

So this would make one fp from KAV engine plus another one from Norman. That's still few. In order to get "many" false positives, there must be another heuristic engine at play...

 

Re: AV-Comparatives June Results (Retrospective / Proactive Tests)

Orion I think is the 3rd heuristic engine

 

Re: AV-Comparatives June Results (Retrospective / Proactive Tests)

I agree that FPs are serious for many users, including myself. However, to take detection and FP together, is there any usefulness to having a ratio of detections vs FP?

Not sure if that would be meaningful or not.

Regards,
Jerry

 

Re: AV-Comparatives June Results (Retrospective / Proactive Tests)

we all knew f-secure would outclass kaspersky in this test

but im sure kaspersky will fight back when their results for v7 arrive,

but only time will tell ... so its just, if's and but's until its done and ready.

 

Re: AV-Comparatives June Results (Retrospective / Proactive Tests)

FPs do matter, but not for all. For example if your PC is mission/time critical, or in case someone is running a small/medium business, then you probably wouldn't like a lot of FPs appearing at the same time. Even if the proactive protection is good, one could say the amount of false detections reported makes it not worth it anyway. So, if you are going to take an AV based solely on AV-comparatives' retrospective tests, then you wouldn't buy AVG because it does not provide good heuristics, you wouldn't buy Avira because it has lots of FPs which would potentially deter your/your employees' work, you wouldn't buy BitDefender for the same reasons.

In the end, informed decisions can only be made by looking at detailed test reports of both the on-demand and retrospective tests. And the dependence/importance of FPs will vary depending upon whether you prefer paranoid protection or whether you want a decent product that will keep your PC safe without bugging you too much about it and not interrupting your work, or you want a compromise between the two.

 

Re: AV-Comparatives June Results (Retrospective / Proactive Tests)

Hi Firecat,
That makes sense to me.
Thanks,
Jerry

 

Re: AV-Comparatives June Results (Retrospective / Proactive Tests)

Or use both:
- A paranoid scanner at the gateway.
- A scanner with low FPs/good detection at the hosts.

 

Re: AV-Comparatives June Results (Retrospective / Proactive Tests)

there is a big error. will change report and tables very soon.
no, i just rechecked, there is only 1 FP for f-secure. the one of norman does not occur in f-secure.

 

Re: AV-Comparatives June Results (Retrospective / Proactive Tests)

There is a small problem in the report:
"Number of false positives found:
2. F-Secure 1 (very few FP)
9. F-Secure 13 (many FP)"

 

Re: AV-Comparatives June Results (Retrospective / Proactive Tests)

IBK, curiously, how many clean files where in the FP test-set, it doesnt state it

 

Re: AV-Comparatives June Results (Retrospective / Proactive Tests)

Praise for F-secure,

so it gets advanced at least then

 

Re: AV-Comparatives June Results (Retrospective / Proactive Tests)

everything corrected/improved now. thanks for pointing to it!
i do not state how many clean files are in the clean set, as it is meaningless.
kaspersky 7 is better than f-secure. the test results of v7 may be released sooner (decision depends from KL).

 

Re: AV-Comparatives June Results (Retrospective / Proactive Tests)

why is it meaningless?

if there are only 50 files, then i agree..... 18 / 26 / 36 is bad, but if there are loads, then it makes the FP results meaningless

it seems that anything over 10 is RED, and gets lowered right?

---------
dang, 12% defragmentation on my ipod

 

Re: AV-Comparatives June Results (Retrospective / Proactive Tests)

Yeah I agree with IC that giving BitDefender the same rating as AVG is ridiculous when it has 6 times the detection and less fps

 

Re: AV-Comparatives June Results (Retrospective / Proactive Tests)

yes .
@MalwareDie: also that has been corrected

 

Re: AV-Comparatives June Results (Retrospective / Proactive Tests)

lol, the more i think about all this... the more it puzzles me,

time for bed i think

hard days work and a weekend to go to

 

Re: AV-Comparatives June Results (Retrospective / Proactive Tests)

I just checked the report again. It is much better now.

 

Re: AV-Comparatives June Results (Retrospective / Proactive Tests)

i was hoping the kav heristics where better than the f-secure ones.
cant wait till KL let you release that.
lodore