Some new features in the coming PCSL's September Test

[pcslinfo]

Hello everyone,
I am Jeffrey from PC Security Labs. After discussing test methodology with some experts in the AV industry and also in our internal team, I will make the following improvement in the coming September PCSL's Test.

a. Dynamic false positive test will be added.
As you know, now the test contains static malware scanning test, dynamic malware block test and static false positive test. While, without dynamic false positive test ,PCSL Total Protection Test is not a complete one. And every time's test, I will pick up 200 representative common software(the number is not yet finally decided) to test the dynamic false positives.

b. how to deal with the classical HIPS
In the dynamic threat block test, if I run a threat and in the first step or the beginning several steps, it can not point out a threat is a threat(e.g. only say it is suspicious or tell you to choose the decision), then it means it failed to detect this malware. While in the dynamic false positive test, if it only says it is suspicious or tell you to choose the decision, it does not lead to a false positive.

c. Malware List Project
As the rapidly increasing threat numbers, how to make the test reasonable is a big problem. And an ordinary consumer can not meet every threat so we have to choose the most prevalent threat samples to make the final list. So there comes the Malware List Project and I have started beta running of the project and it seems good. I will ask the vendors to provide me not only the most prevalent sample itself but also the information attached with it(Prevalent level, the behavior, first seem time, etc). The samples coming from the vendors will make up 60-70% of the final Malware List package to be used in the test. And the other 30-40% will come from PCSL's own research. Those samples having higher prevalent level or more detailed information will have priority to be added into the final package.
Since then, there are more than ten security vendors starting to report their threat research result to PC Security Labs and I also welcome more vendors to join this project to build a more reasonable prevalent sample package to better reflect how their products can protect us ordinary end-users.


Thank you for all the readers and all the experts to help me improve the test.
Jeffrey

[vijayind]

Hi Jeff,
One more suggestion. I see that in your test a dozen or so apps get excellent 5 stars rating. But some detect 99% others 95%. Clearly the higher scoring av deserves some credit.

So could from now on at rank at the bottom of your award. So X antivirus is best so its award is "PCSL Excellent 5 stars" with rank 1 written on the bottom. Antivirus Y came second so it also gets the same "PCSL Excellent 5 stars" award but with rank 2 written underneath.

Regarding your new test scheme. I think its evolving to the changing products which use dynamic components and HIPS. So good work there Kudos !!

[pcslinfo]

Originally Posted by vijayind

Hi Jeff,
One more suggestion. I see that in your test a dozen or so apps get excellent 5 stars rating. But some detect 99% others 95%. Clearly the higher scoring av deserves some credit.

So could from now on at rank at the bottom of your award. So X antivirus is best so its award is "PCSL Excellent 5 stars" with rank 1 written on the bottom. Antivirus Y came second so it also gets the same "PCSL Excellent 5 stars" award but with rank 2 written underneath.

Regarding your new test scheme. I think its evolving to the changing products which use dynamic components and HIPS. So good work there Kudos !!

I will discuss your advice in my team and thank you for your consideration
For the hips or dynamic components rule, it is not completed and I need more suggestion on that.

[andyman35]

It's admirable that this group are making efforts to test these products in a representative fashion,keep up the good work!

As for judging the HIPS components,the primary factor in determing a pass or fail should be how the warnings would likely be interpreted by an average user which is the way you appear to be going.

[subset]

Originally Posted by pcslinfo

b. how to deal with the classical HIPS
In the dynamic threat block test, if I run a threat and in the first step or the beginning several steps, it can not point out a threat is a threat(e.g. only say it is suspicious or tell you to choose the decision), then it means it failed to detect this malware.

So to pass or fail a test will be a question of semantics?
And the wording winner is...

It's called dynamic threat block test, so why don't you test according to this description
Simply allow the malware to run, block all other prompts and if something malicious slips through, the HIPS has failed the test.
That's a dynamic threat block test.

Cheers

[pcslinfo]

Originally Posted by subset

So to pass or fail a test will be a question of semantics?
And the wording winner is...

It's called dynamic threat block test, so why don't you test according to this description
Simply allow the malware to run, block all other prompts and if something malicious slips through, the HIPS has failed the test.
That's a dynamic threat block test.

Cheers

If the behavioral detection moudle is intelligent and automatic, then I will judge the result through the real effect however it says suspicious, threat, high risk.

If the behavioral detection moudle is not intelligent, multi-steps or need users to decide what to do with unclear notification, then an ordinary user will trend to choose permit(as usually most of the softwares they meet is clean) and then lead to a bypass of the malware.

I will use equal principal to be used both on clean file and threats

Thank you for your suggestion.

[vijayind]

Originally Posted by pcslinfo

I will discuss your advice in my team and thank you for your consideration
For the hips or dynamic components rule, it is not completed and I need more suggestion on that.

For HIPS, I am not sure you will get a pop-up that says threat found. Most HIPS will notify you when they see some driver/rootkit being installed or protected files/registry being manipulated. So installing your new video card software or getting infected with a rootkit will show almost alike pop-up in a HIPS.

Average Joe and Moe, would probably see the popup. If it clearly shows that the app doing the naughty stuff is a suspicious named app they would DENY it, else say OK. Only first few popups would get the deserved attention and would be clicked DENY. If more than (say) 3 popups come up for a malware, most likely average joe is going to click ALLOW and check the create rule option.

I know all of this is quasi. Better check up with AMTSO, they may have some guidelines for HIPS testing.

[pcslinfo]

Originally Posted by vijayind

For HIPS, I am not sure you will get a pop-up that says threat found. Most HIPS will notify you when they see some driver/rootkit being installed or protected files/registry being manipulated. So installing your new video card software or getting infected with a rootkit will show almost alike pop-up in a HIPS.

Average Joe and Moe, would probably see the popup. If it clearly shows that the app doing the naughty stuff is a suspicious named app they would DENY it, else say OK. Only first few popups would get the deserved attention and would be clicked DENY. If more than (say) 3 popups come up for a malware, most likely average joe is going to click ALLOW and check the create rule option.

I know all of this is quasi. Better check up with AMTSO, they may have some guidelines for HIPS testing.

yes, hips is difficult to test, while as there are dynamic false positive test along with the dynamic threat block test, so the equal principal can be
guaranteed.

Some hips will have color ranking system, and my action will based on a normal user's decision that if they think this will infect their machine, I will choose deny.

PS, I am a member of AMTSO

[Sputnik]

@Jeffrey
I will look into your new methodology asap. Is there any news about possible new vendors getting added? In our labs we see a massive detection increase by AVG/Grisoft the last 6/8 weeks (mainly SHeur2). Also Microsoft's generic signatures stand out on many 0day threats. It would be nice to have these (and others) in the testing field. If the vendors agree of course.

(Ps. As promised, I will still PM you.)

[NobleT]

The More overall testing ,the more better development will be gotton for PCSL,It means that the PCSL testing will be get acceptted for more and more people.
expecting! it

[pcslinfo]

Originally Posted by Sputnik

@Jeffrey
I will look into your new methodology asap. Is there any news about possible new vendors getting added? In our labs we see a massive detection increase by AVG/Grisoft the last 6/8 weeks (mainly SHeur2). Also Microsoft's generic signatures stand out on many 0day threats. It would be nice to have these (and others) in the testing field. If the vendors agree of course.

(Ps. As promised, I will still PM you.)

hmmm, maybe it is time to release a new methodology as PCSL Total Protection Test is basically complete

There are 5 vendors in the internal test and i will check if some of them want to take part in the public test.

For AVG and MS, through my daily research, their proactive detection method(heur detection, generic detection) is good and I also wanna see they are in the public comparative test list.

Thank you Sputnik