PCSL Testing Report 2009 NO.1

[Lawliet]

PCSL 2009 Total Protection Testing Report NO.1 is release now
Please to our official website
http://www.pcsecuritylabs.net/news.php?readmore=20


And Panda Blog
http://research.pandasecurity.com/archive/Panda-participates-in-new-AV-comparative.aspx
-{ Quote: "Since a few months ago we've started participating in a new AV comparative test from PC Security Labs called Total Protection Testing. It's a pretty kewl test since, as opposed to other AV comparatives out there, PC Security Labs has a very interesting testing methodology that takes into consideration:

* Freshness of malware samples. Only the newest samples from the previous month are tested, not year old samples.
* Static detection using traditional signature files, very similar to what other AV comparative testers are doing.
* Dynamic (behavioral) detection of malicious running processes. Only a handful of professional AV testers are doing this.
* Cloud-based detection such as Panda's Collective Intelligence. As far as I know PCSL is the first AV tester with a methodology that takes this type of technology into account.
* False positive testing. Global scores are lowered on each false positive.

All-in-all a very complete testing methodology that gives a broad view of the global performance of different anti-malware solutions. It's no surprise that PC SecurityLabs has recently joined the AntiMalware Testing Standards Organization (AMTSO).

I'm glad to report that Panda has achieved an "Excellent" score in each of the three tests we've participated in so far.

I'm glad to report that Panda has achieved an "Excellent" score in each of the three tests we've participated in so far.

Total Protection Testing reports from PCSL can be downloaded directly from the following locations:

* PC SecurityLabs Total Protection Testing 2008/11
* PC SecurityLabs Total Protection Testing 2008/12
* PC SecurityLabs Total Protection Testing 2009/01

The tests are performed on a monthly basis, so make sure to visit PC Security Labs every now and then to get the latest results!" }-

[firzen771]

Yet again, good job to the people at Avira, twister did really well as well, that was nice to see

[Coolio10]

Wow, good job to filseclab heuristics. No fps aswell!

[Fuzzfas]

Way to go Twister, little fella! :argh:

-{ Quote: "Wow, good job to filseclab heuristics. No fps aswell!" }-

It does have heuristics in the on demand scan, but most is due to the FDD (behaviour blocker). Even on default settings, it's quite aggressive. It's not a surprise that it works well, it is known to score always well in these tests and to complement the signatures in a good percentage.

The absense of false positives is pure luck. Believe me, i know! ;D Just a few days ago i submitted once more the binary of Abiword that as in every new release, is detected as Trojan Zhelatin. (It's now fixed...again). So... it has no false positives... Hmm... Probably no other legitimate application was installed on the test PC. So all it could find was the real malware! :D

In the next version there will be improvement in both engine and FDD.

[Fuzzfas]

-{ Quote: "Yet again, good job to the people at Avira, twister did really well as well, that was nice to see" }-

Avira is getting boring... It spoils all the mystery being always first! :D Well, Emsisoft got the same percentage...

Both are taking the crown off KAV's hands in most tests out there.

On another note, what has happened to TrendMicro?! It used to be very famous. Up to a few years ago, in most motherboards they were giving PC-Cillin on a CD.

[MalwareDie]

Way too many 99%'s and the sample size is too small for me to take this seriously.

[Fuzzfas]

-{ Quote: "Way too many 99%'s and the sample size is too small for me to take this seriously." }-

Just between me and you, i don't trust any AV test, but they 're good for product promotion. ;) So, yay Twister! ;D

There is one half-truth Panda is saying here:

* Freshness of malware samples. Only the newest samples from the previous month are tested, not year old samples.

"Fresh" = previous month. If they were more fresh, most wouldn't arrive at 80% most probably. 90+s are good for the vendor. It promotes a safety feeling for the customer. Year old samples are good in that too, because most likely everyone detects them, so they all start on a good basis, which is good for business. The "very fresh" are the real problem that make the difference...

[TonyW]

I'm more surprised at Panda's spelling of "kewl" in their blog.

[pcslinfo]

-{ Quote: "Yet again, good job to the people at Avira, twister did really well as well, that was nice to see" }-

We use default setting as Filseclab suggested
Default setting will lead to some reduction of detection rate and less false positive.;D


-{ Quote: "Way too many 99%'s and the sample size is too small for me to take this seriously." }-
Our testing is to test the comprehensive competence of a single antivirus testing. As you see, we run all the samples each scanner missed in the static testing to simulate the real infection may occur during the normal use of PC users. And we sort the sample by prevalent level and every time we use fresh samples to hold this testing. And we judge the sample malicious or not by behavior analysis to ensure the functionality of each sample:)


-{ Quote: ""Fresh" = previous month. If they were more fresh, most wouldn't arrive at 80% most probably. 90+s are good for the vendor. It promotes a safety feeling for the customer. Year old samples are good in that too, because most likely everyone detects them, so they all start on a good basis, which is good for business. The "very fresh" are the real problem that make the difference..." }-

Our testing is focusing on the comprehensive competence of a single antivirus testing. The very fresh samples you mentioned are used to hold the response time testing;D

[Fuzzfas]

-{ Quote: "We use default setting as Filseclab suggested
Default setting will lead to some reduction of detection rate and less false positive.;D " }-

Still, it is a miracle. Probably you didn't have a good number of legitimate applications around. ;D Don't get me wrong. I am fan of Twister and i love it. But as false positives are concerned, even with default settings, well...


-{ Quote: "
Our testing is focusing on the comprehensive competence of a single antivirus testing. The very fresh samples you mentioned are used to hold the response time testing;D" }-

I am no big fan of AV tests, but it's fine by me and i hope you will will create a reputation. The detection rates in absolute numbers, depend much on what samples you use (age is one important factor). So, the more AV tests out there, the more reliable the results can be, for the relative detection rate of AVs. Meaning i do believe Avira is 1st, because in most AV tests it is 1st. Now, the 99% is another story, depends much on sample age, malware geographic origin etc.

So, keep up the good job! I particularly like the fact that you also do dynamic testing and pubblish in detail that result too. This way i could evaluate how good FDD is.

[pcslinfo]

-{ Quote: "So, keep up the good job! I particularly like the fact that you also do dynamic testing and pubblish in detail that result too. This way i could evaluate how good FDD is." }-

I will try my best to perfect our testing(better methodolody, larger clean file database, more representative malware samples) and thank you for your consideration and advice:)
Have a nice day and also Good Luck;D

[Fuzzfas]

-{ Quote: "I will try my best to perfect our testing(better methodolody, larger clean file database, more representative malware samples) and thank you for your consideration and advice:)
Have a nice day and also Good Luck;D" }-

I am sure you will. I also enjoyed your malware links in your forum. I read you closed access to the pubblic to the malware database, i hope you will keep posting the web based malware in the forum. It's good for enthusiasts to be able to test their defenses once in a while.

Also thanks for including Twister. You 're the only one that did so. I am fond of this underdog and most probably, being unknown and with small userbase/lifetime license, they can't afford paying the $$$ that well known magazines or tests require.

[pcslinfo]

-{ Quote: "I am sure you will. I also enjoyed your malware links in your forum. I read you closed access to the pubblic to the malware database, i hope you will keep posting the web based malware in the forum. It's good for enthusiasts to be able to test their defenses once in a while.

Also thanks for including Twister. You 're the only one that did so. I am fond of this underdog and most probably, being unknown and with small userbase/lifetime license, they can't afford paying the $$$ that well known magazines or tests require." }-

As there are too many things while the new year come:wacko: I will redesign the website in the Chinese Spring Festival and probably restart the web malware link at that time:)

[Fuzzfas]

-{ Quote: "As there are too many things while the new year come:wacko: I will redesign the website in the Chinese Spring Festival and probably restart the web malware link at that time:)" }-

Ah, new design too! Best wishes! And, yes, please do restart the web malware link then! It's very helpful! :thumb:

[GES/POR]

-{ Quote: "As there are too many things while the new year come:wacko: I will redesign the website in the Chinese Spring Festival and probably restart the web malware link at that time:)" }-

Can you include f-prot n prevx edge in your tests?

[Saraceno]

Nice forum too. Links to allow the testing of 'shifty stuff'! :lurking: :D

Your testing is great as well, launching the file the same as a user would experience in the real environment. Much better than a simple right-click on a specified folder and then scan.

[pcslinfo]

-{ Quote: "Can you include f-prot n prevx edge in your tests?" }-

We invite the AV vendors to join our testing program, and after the AV vendor offically announces to participate, we then add them into the testing list. When there is new AV vendor who accept the invitation, I will post it in my homepage and let everyone knows:)
Thank you for your consideration and have a nice day
Regards
Jeffrey


-{ Quote: "Nice forum too. Links to allow the testing of 'shifty stuff'! :lurking: :D

Your testing is great as well, launching the file the same as a user would experience in the real environment. Much better than a simple right-click on a specified folder and then scan." }-
I will try my best;)
Thank you and enjoy yourself:)

[TonyW]

-{ Quote: "Links to allow the testing of 'shifty stuff'! :lurking: :D" }-I don't think that's a good idea for the same reason forums like this one don't allow links to malware.

[Fuzzfas]

-{ Quote: "I don't think that's a good idea for the same reason forums like this one don't allow links to malware." }-

This forum doesn't also allow AV A vs AV B, but it allows firewall A vs firewall B, antimalware A vs antimalware B, any other software A vs any other software B, hardware A vs hardware B etc.

People with bad intentions know exactly where to find malware (they have dedicated sites and fora), they don't wait for PCSL to put 10 infected sites to find malware. On the other hand, PCSL is a test lab, not a forum (the forum is merely an extra) and having a few links with infected sites can be an easy way for users that don't want to go to malware sites to find some malware. You click on your own risk. It's not that they try to shove you malware down your throat.

The other week i wanted to find malware for my own testing. I had to go to the "dark" side, in sites that i didn't know if they had some exploit for the unaware visitor. It's much safer to know that i can find a few samples from a "secure" site than wondering from one malware site to another.

If "Pcslinfo" decides to eliminate those links too, i only hope that it can make them accessible to whoever wants to via password. In this case click-happy users that don't know what they are doing will have less to complain.

[TonyW]

-{ Quote: "If "Pcslinfo" decides to eliminate those links too, i only hope that it can make them accessible to whoever wants to via password." }-If they're going to allow links to malware, that might be a better approach rather than have them openly accessible. Yes, I know sites and forums exist where such things are, but they're not freely advertised here mainly for safety reasons.

[Fuzzfas]

-{ Quote: "If they're going to allow links to malware, that might be a better approach rather than have them openly accessible. Yes, I know sites and forums exist where such things are, but they're not freely advertised here mainly for safety reasons." }-

They did have a malware pool but they password protected it and made accessible for testers. Now the only "freely" accessible "malware source" are a few links in the forum, that point to infected sites. But they are quite old now, most AVs will detect them.

I know that here such links are forbidden and it's ok. This is a forum and they have their policy. PCSL is a more particular site though, i think that a few links to infected site can be allowed to exist for those who want to test their defenses. On their own risk (put a nice warning sign).

Or password protect that too, but not limit it to AV researchers. I prefer going to PCSL to find some malware than to malware fora or sites.

[Saraceno]

But at the same time, listing malware links shows you which sites are continually pumping out software loaded with 'problem files'. For example, lot of links to the brothersoft site. What I learned? I wouldn't probably download software from that site.