PCSL Total Protection Testing (2009 May's report)

[pcslinfo]

Hello everyone,
I have finished the new May's report for PCSL Total Protection Testing.
http://www.pcsecuritylabs.net/news.php?readmore=29

And you can download report from here:
http://www.pcsecuritylabs.net/document/2009May.rar

I will be here discussing the details with all of you and your comments will be of importance to me;D

Jeffrey

[Thankful]

Creating a readable report would be a nice start. I can't view the results in either Firefox or I.E.

[vijayind]

Thks for the results, Jeff.

A suggestion: Since 12 products are almost neck-n-neck for the 5 star rating. Could you add details of what malware was missed by which product. So that one can evaluate better the difference between the products.

[subset]

There should be an explanation on the website what exactly is...
- Detection Rate in Static Testing
- Total Detection Rate
- Final Score
- Award

But not... Marking system=（A+B）/C*100-lg(D+1) :argh:

Cheers

[progress]

A2 and the false positives - a neverending story ::)

[pcslinfo]

-{ Quote: "Thks for the results, Jeff.

A suggestion: Since 12 products are almost neck-n-neck for the 5 star rating. Could you add details of what malware was missed by which product. So that one can evaluate better the difference between the products." }-

You can check that in detection rate in the static testing

-{ Quote: "There should be an explanation on the website what exactly is...
- Detection Rate in Static Testing
- Total Detection Rate
- Final Score
- Award

But not... Marking system=（A+B）/C*100-lg(D+1) :argh:

Cheers" }-
Hmmm, I will add manual in the package next time:)


Thank you for your suggestions. Good Luck;D

[raven211]

Hey, Panda did fair very well here (well? freakin' first ;D). Would you say that it's in the top by now and that their proactive features are really improving? P-CAV will probably be quite dangerous once Beta 2 is released. ;D AV-Comparatives testing of Panda (ProtectStar reports) have been different when it comes to Panda... :)

[MaB69]

Hi,

It seems that i am the only one unable to load PCSL Website, using OpenDNS as DNS provider

209575

I have no idea what's wrong :-\

Will try later but thanks for your work

Regards,

MaB

[1000db]

From those results it seems to illustrate difference in detection methods. For example, Avira and A2 seem to focus mainly on fingerprints or definitions while others like Bitdefender and Twister make heavy use of dynamic (non-signature detection as I understand it) detection. BTW what's up with Twister only having 1 FP!

[Fuzzfas]

-{ Quote: "From those results it seems to illustrate difference in detection methods. For example, Avira and A2 seem to focus mainly on fingerprints or definitions while others like Bitdefender and Twister make heavy use of dynamic (non-signature detection as I understand it) detection. BTW what's up with Twister only having 1 FP!" }-

LOL! Lately there is a reduction on Twister's FP. I myself have seen none for some months. They must be working on that. I 've also seen a reduction on the on-demand scores. My guess is, they have neglected signature adding, in order to put more manpower to the new version. As a result, the FDDS (the behavious blocker), has increased its share of detection (in the above test, the FDDS accounts for roughly 25+% of Twister's total detection!).

[Sputnik]

Thanks Jeffrey! :thumb: Will check out the report asap.

[pcslinfo]

-{ Quote: "Hi,

It seems that i am the only one unable to load PCSL Website, using OpenDNS as DNS provider

209575

I have no idea what's wrong :-\

Will try later but thanks for your work

Regards,

MaB" }-

Sorry to hear that, DNS problem of your ISP????

-{ Quote: "Thanks Jeffrey! :thumb: Will check out the report asap." }-

Waiting for your suggestions:-)

[Einsturzende]

Kaspersky was tested in automatic mode, isn't it?

[pcslinfo]

-{ Quote: "Kaspersky was tested in automatic mode, isn't it?" }-
it is:)

[vijayind]

-{ Quote: "You can check that in detection rate in the static testing
" }-
Hi Jeff,
That's just a number.
If you could display the exact malware missed. Then one can evaluate how good the protection of AV is. For ex: If ACME Anti Virus misses say conflicker.c sample then even though it has a 95% detection rate, its probably not all that good.

[Einsturzende]

-{ Quote: "it is:)" }-
in proactive mode you will have complete power of KIS HIPS and detection/protection will be ~100% :o, you should consider running it in proactive mode for ("missed") "on execution" test next time... ;)

[pcslinfo]

-{ Quote: "in proactive mode you will have complete power of KIS HIPS and detection/protection will be ~100% :o, you should consider running it in proactive mode for "on execution" test next time... ;)" }-

Actually proactive mode is for advanced users, auto mode is for most of the users. Normanl users actually can not just whether it is malicious or not, anyway, through the test, KIS's auto mode is good and strong enough:)

[pcslinfo]

-{ Quote: "Hi Jeff,
That's just a number.
If you could display the exact malware missed. Then one can evaluate how good the protection of AV is. For ex: If ACME Anti Virus misses say conflicker.c sample then even though it has a 95% detection rate, its probably not all that good." }-
You mean the exact missed samples' detailed information? But if one missed a lot, how can it be displayed??? ::)

[Einsturzende]

-{ Quote: "Actually proactive mode is for advanced users, auto mode is for most of the users. Normanl users actually can not just whether it is malicious or not, anyway, through the test, KIS's auto mode is good and strong enough:)" }-

fair enough, but should be noted there is "reserve power" in KIS and protection can be further enhanced by adjusting KIS settings :)

EDIT:
BTW. this test is example in which way professional tests should go, if it is possible add for next testing small amount of "malware from the source", complete vector of infection to see particular antimalware solution in real environment, e.g. from malicious url to malware execution and disinfection...
many thanks for testing

[pcslinfo]

-{ Quote: "fair enough, but should be noted there is "reserve power" in KIS and protection can be further enhanced by adjusting KIS settings :)" }-

That's why they offer proactive mode for the advanced users like you:)

[vijayind]

-{ Quote: "You mean the exact missed samples' detailed information? But if one missed a lot, how can it be displayed??? ::)" }-
Well, I thought it wouldn't be too hard for you. Since you seem to number your malware files with unique names. And if the AV is set to scan and delete/quarantine you will only be left with undetected samples in your malware folder.

So just see the files left and list out corresponding names. I think you could even right a script for it ( using grep ).

At this point, I am just making assumptions. So if your load is high, if possible list the missed malware you think were important. So that a reader can differentiate between the performance of all AVs getting 95+%.

[trjam]

well actually F-Secure should bump up a slot with 2010 and the Bitdefender engine instead of Kas 6. Along with their own engines to compliment it, looks like good times ahead.

[Pleonasm]

Jeffrey, an idea to consider: It would be interesting to extend the analysis to pairs of anti-malware products, under the assumption that increasingly multiple security products are utilized for protection (e.g., a primary real-time anti-virus coupled with a secondary product for on-demand, manual scanning). Such an analysis would answer the question, “Which two products should I use in combination to maximize protection (and to minimize false positives)?” Obviously, not all products “play well” in combination with one another, so that practicality might limited the number the pairs examined.

Question: What is the process by which vendors are selected for inclusion in your analyses?

[kdcdq]

Jeff, thanks for your work and I enjoyed reading your test results.

I do have a question, though:

Is there a reason that you did not test AVG? It would have been nice to see how AVG compared to the other AVs tested, me-thinks.

Thanks in advance.

[Ford Prefect]

-{ Quote: "Well, I thought it wouldn't be too hard for you. Since you seem to number your malware files with unique names. And if the AV is set to scan and delete/quarantine you will only be left with undetected samples in your malware folder.

So just see the files left and list out corresponding names. I think you could even right a script for it ( using grep ).

At this point, I am just making assumptions. So if your load is high, if possible list the missed malware you think were important. So that a reader can differentiate between the performance of all AVs getting 95+%." }-

Hi Jeff,
thanks for your report.
A list of important false positive detected files would be interesting, too.
Regards,
Ford

[Pinga]

Why are Avast, AVG, and ESET not among those who 'officially take part' (sic)? What were the inclusion criteria?

[pcslinfo]

-{ Quote: "Well, I thought it wouldn't be too hard for you. Since you seem to number your malware files with unique names. And if the AV is set to scan and delete/quarantine you will only be left with undetected samples in your malware folder.

So just see the files left and list out corresponding names. I think you could even right a script for it ( using grep ).

At this point, I am just making assumptions. So if your load is high, if possible list the missed malware you think were important. So that a reader can differentiate between the performance of all AVs getting 95+%." }-

Got your point, you wanna know what they missed for every single scanner, right?
I will discuss with my team member and thanks for your suggestion:)



-{ Quote: "well actually F-Secure should bump up a slot with 2010 and the Bitdefender engine instead of Kas 6. Along with their own engines to compliment it, looks like good times ahead." }-


When 2010 is finally released, we will replace the old 2009 to the new one.

-{ Quote: "Jeffrey, an idea to consider: It would be interesting to extend the analysis to pairs of anti-malware products, under the assumption that increasingly multiple security products are utilized for protection (e.g., a primary real-time anti-virus coupled with a secondary product for on-demand, manual scanning). Such an analysis would answer the question, “Which two products should I use in combination to maximize protection (and to minimize false positives)?” Obviously, not all products “play well” in combination with one another, so that practicality might limited the number the pairs examined.

Question: What is the process by which vendors are selected for inclusion in your analyses?" }-

Sometimes it will cause conflict to install two software at one time, and the result will be different if you choose who will be the main scanner to scan or guard.
Unlike trustport, F-Secure, G DATA, they have ability to combine them smoothly in one single product, so such problem above will not happen while using.
Anyway, your idea is intesesting and thanks for your consideration;)


-{ Quote: "Hi Jeff,
thanks for your report.
A list of important false positive detected files would be interesting, too.
Regards,
Ford" }-
More details about the FP will be added in July's Test;D Thank you

-{ Quote: "Why are Avast, AVG, and ESET not among those who 'officially take part' (sic)? What were the inclusion criteria?" }-

We need authorization for AV vendors to allow us to test, so if I got official AGREE, I will surely add them into the testing list.:)

[NobleT]

someone also have a ID in this forum mm i have guessed~~:P

[Pleonasm]

-{ Quote: "We need authorization for AV vendors to allow us to test" }-
Pcslinfo, can you please explain this further? I do not understand how a company could prevent you from performing and reporting upon a product test.

Thanks.

[pcslinfo]

-{ Quote: "Pcslinfo, can you please explain this further? I do not understand how a company could prevent you from performing and reporting upon a product test.

Thanks." }-
Hi Pleonasm,
I think to perform a good test, it is a must to get a good communication channel between vendors and us. So it will come easy when there is problem while testing, how to set the parameters and etc.
So we just contact vendor and let them know how we perform the test, how we filter the samples and then they agree us to test and finally we add them into the testing list.
So there comes why we need to discuss with vendor before test and why we need authorization.

Many thanks for your time and advice:)