New Report, New Website and Greetings from PCSL

Do I need glasses, am I still suffering/toxicated from new year celibration or is Avast not tested?

Best wishes for 2010 everyone

 

No, it's not tested. As the tester said, in order to test a software, he needs explicit approval from the vendor.

 

Thx that is a pity and a relief (it is not me)

 

Well its a shame that avast, norton, eset weren't approved to be tested from their respected vendors

 

For this, I think whether to take part in test held by test lab is vendor's choice. To hold a good test, there should be good communication bwtween test lab and vendors, as no one can understand the security products(the technology they use etc.) better than vendors themselves. From my side, I never force any vendor to take part in any time of test, and I think to improve test to better reflect the real status is my first thing to do. Through my effects, seems that most of the players in the industry had taken part in my test whatever public or internal So here, I want to say thanks to all the friends in the AV industry and also the readers as you guys

 

As what I see in the report, there are both "Static" and "Dynamic" detections, so the rate(%) which you see is the total(Static+Dynamic) result only.

 

look nice and beautiful

 

Again to me the detections seem off, yes its a smaller sample set but if I remember right that's why AV Corp uses such a big sample set for these kind of tests. Is to make the results more balanced.

Again thanks for the test but I think I will stick to AV-Test and AV-Corp.

 

Let me explain why we use small scale of sample set.
A. Every time we fresh the sample so the detection will not be superimposed.

B. We are trying to make a new prevalent sample set to be used for test. The draft is: e.g. every vendor provide us 200 most prevalent sample per month, then we receive 4000 samples from 20 vendors and then we add 2000 prevalent samples from PCSL's own monitor system. So this sample set will be the most prevalent samples that infected the largest number of PCs. The threat who infected 1 million pc and the threat who infected 100 pc do not have the same meaning definately.

C. We have added dynamic test into the total test, that means we have to execute every sample missed manually in static scan test, so we can not expand the sample set to a much larger scale.

Hope my words help you to understand why we use small scale of the test

 

Wait I'm hoping I am misreading this. Your saying you get most your samples from AV vendors ? If this is true then the darn thing is already detected by that vendor how does this even remotely help test a product. If I misunderstood you please correct me.

 

First I have to tell you the following premises:

A. AV Test lab do not create virus to test AV products
B. AV Test lab not only collect samples via own ability but also via AV vendors. Please also take a look at wildlist.org, who is the most common sample set used by vb or wcl, the wildlist is reported by vendors, at least most of them.
C. We use our own collection into the sample set and the proportion is not a small amount(30-40%).
D. No One can better monitor the infection status except vendors. So a mixture prevalent samples package attached with PCSL's samples via our own monitor system will be a reasonable sample set to refelct the real infection status.
E. The samples used for test is not the whole collection of PCSL's database, only very small proportion, so maybe you think that the samples we use for test is the whole package we collect, sorry for my words to lead you misunderstand.

I will follow your topic if you still have questions. And for my side, really thank you for your consideration.

 

All in all then the sample set is really much lower then as most of the samples are already detected by the vendor that reported them. Just seems like an extremely stupid way to do a test then again maybe that's just my view. As for the VB using the same kind of sample set I also don't set any faith in that test ether. If you don't pass that one there is something seriously wrong with your AV product.

 

I catch your point and let me say something:

A. PCSL Total Protection Test is a comprehensive test to test AV products. As you see, we have dynamic test(also test malware and clean install package), so that you will see the perfomance when the malware is not detected and collected by a AV product. And also, to have a short term collective ability(e.g. two months) is also an index to judge a AV products. To test samples that are totally new to AV vendors is unpractical, when AV labs are catching samples(e.g. from web-base malware), AV vendors also have the ability to do the same thing. And to create malware to test, at least in China, is not legal.

B. Response time test, Heur test and cloud test are method to test vendors' ability while facing new threats, and I strongly recommand you take a look at them and I think you will like those test. PCSL is developing those tests and we have finished some demo test, and I will show you when the system is mature enough.

I will follow your questions here.

 

av-c have been known to recieve samples from AV vendors too.

 

Yes but not over half, that is what I was clarifying on. It makes the "Real Unknown" sample set much lower and the test alot easier to pass that's all I was pointing out.

 

how do you know?

i was under the impression that 'most' of av-c's samples were given to them in one way or another.

 

I guess in the end it would just be better to have IBK Post on this subject as I really don't have the answer to that one. If he reads this and chimes in we will go from there.

 

AV-Comparatives states:

AV-Comparatives have various sources from which it obtains samples. Like anti-virus vendors, we also use various traps and a large quantity of honeypots from all over the world, as well as samples downloaded from malware downloaders and infected websites. Furthermore, we get samples from the field which were collected by us or our partner companies (e.g., computer repair/cleaning services) on infected PCs belong to home users and/or small/medium business companies. We also get samples from various online scanning services and (single and large) submissions from visitors to our website, as well as various organizations that collect malware (internal and public security forums, honeypot projects, anti-malware initiatives, and so on). In order to have a test-set that is statistically valid and as large and representative as possible, AV-Comparatives also accepts samples from (security) vendors. Currently, samples submissions from about a dozen vendors are included in our tests and nearly dozen more vendors which are not included in our tests also contribute.

Source: Methodology & FAQs​

For its most recent dynamic test, note that AV-Comparatives independently selected its test cases: “The URLs were collected by using our own in-house crawler; to avoid bias, we did not use any publically available services which deliver malicious URL feeds” (source: Whole Product Dynamic Test, December, 2009). This approach certainly makes the most sense to me.

I fail to see any reason why a testing organization needs authorization and participation from an anti-malware vendor.

 

yep, i agree aswell.

all tests should include all solutions, regardless if permission to test the product has been given, or regardless if a certain company has or wont pay a fee to get tested.

when money to test starts getting transferred, there will always be question marks over the results, it just makes it less 'independant' to me.

 

Let me give you an example,half a year ago I wanted to pre-test the McAfee's cloud detection using their command line scanner but I lack the .DAT file to enable the Artemis function. Do you think if there is not effective communication channels a test lab can know how to realize this? To have good communication can help you achieve a better result, and that is also why I reply to you here to let you have a better understanding of what I mean.

 

I think there is misunderstanding:
I only say
"For PCSL, we only add vendors when they officially tell me that you are willing to take part in the public test." in floor 24.
It is not my words that I say I have to get the permission.

The real flow is:
A. we send invitation to a vendor and vendor says they are willing to participate, then we add.

B. vendor send the willing to PCSL without PCSL's invitation, then we added.

And also, we will discuss with vendors what product they want us to test and send us technology whitepaper for us and introduce their techonology via email or phone to know better about their products. I have mentioned above why there should be good communications between test lab and vendors. No one can better know the security products better than vendors themselves, as no one can be known better than their parents. If test lab do not get fully understanding of the product, the result will have the possibility to be off-course. As if you want to drive a Ferrari, you should get the full parameters,right?

For testing fee of PCSL Total Protection Test, PCSL do not charge any fee from vendors both the test report and award logo. It is not hard to find guys from security vendors here in WSF and everyone can select a person randomly here to verify what I am saying.

hehe

 

Someone has to pay for the testing, however. I don’t object to anti-malware vendors contributing to the funding of a test, but the decision by the testing organization about which products to include in its assessment should not, in my opinion, be predicated upon the vendor providing “consent.”

If you can’t obtain the vendor’s anti-malware software publically, then neither can the readers of the test. And, if the software can’t be obtained by consumers, then the results of the test are unlikely to be of much interest to that audience.

 

In my opinion, a testing organization should seek to mimic, as closely as possible, the actual use of the anti-malware product by a consumer. If a consumer doesn’t have access to a proprietary “full understanding of the product,” then neither should the testing organization.

 

I think this explains why everyone is clustered together. The prevalent samples, are samples that everyone should detect in the first place. Probaby the rest of the 2000 are also "common".

The result is that the samples probably represent "common malware" , that's why everyone scores so well.

On the other side, being so closely clustered together and taking in account the sample set, one must assume that globally, a small percentage difference, in another test with much more samples, including more rare, would translate in big differencies.

So, if you get a 97% here in PCStest, in a test with more rare samples, your detection will be far more away than someone with 98% detection in PCS test.

 

I will answer you the two question:
A. PCSL do not charge fees for PCSL Total Protection Test both the report and the award logo and also there is no obstacle for vendors. So vendors do not need to pay for public test as I mentioned above.

B. .DAT file can enable the artemis function while consumer product has this function by default. Using command line scanner is a good method to hold pre-test, please note that I say "Pre-test", in public test, we use the same software as client use.

Will follow your questions,