Something about testing sample set selection in PCSL Total Protection Testing

[pcslinfo]

Q: Why every month you just choose about ~2000 malware samples to test the security software?
A: Every month I received the samples, I have to follow several steps to pick up the final testing samples from the original sample bed.
a. Un-achieve the sample and delete the duplicated ones.
b. Use PEID to sort the samples and delete the non-PE files.
c. Run all the samples with exe extension in VMware and judge whether they have malicious behavior using the guard of Host Intrusion Prevent System. And then delete the clean, corrupted, non-malicious ones.
d. Scan the malicious samples left with the local multi command line scanner (24 scanners) and then get the characters from the detection name of each scanner such as prevalent level, when coming out, the region and etc. And then we pick out the prevalent and fresh samples through the whole database.
e. Through these procedures, the final left ones is about 2000-3000. And every month we will use new samples and will not use the samples we have ever used in the last month.

Q: How about the samples selection about the auto generation malware such as SWIZZOR family?
A: We will not add these samples into the testing bed, as in some aspects, normal users will not been infected like the other prevalent ones.

Q: Will you add packed normal program into the clean files set?
A: NO. We will not add these kinds of packed normal program, such as packed notepad, packed calculator into the clean files set and we will not add some grey software such as keygen.

Thank you for all of your consideration and wish everyone a nice day;D

[funkydude]

Q. There are 39 AV's on VT, why are so little tested?

Q. "Every month I received the samples" - People send you the samples, that sounds like a pretty untrustworthy way to perform testing?

[i_g]

-{ Quote: "b. Use PEID to sort the samples and delete the non-PE files.
" }-
I don't think PEID is a good tool to judge validity of PE files (or any other, for that matter - other than Windows loader itself); some files reported by PEID as invalid can be started without any problems, and vice versa.

[pcslinfo]

-{ Quote: "Q. There are 39 AV's on VT, why are so little tested?

Q. "Every month I received the samples" - People send you the samples, that sounds like a pretty untrustworthy way to perform testing?" }-

We need some kind of agreement that AV vendors offically take part in our testing and we will not test them before that.
To test a product without any agreement will sometimes cause some problems such as copyright, etc.

Receive means I don't make virus and they are all coming through different channels. And can you tell me what is the trustworthy way in your mind?

Thank you very much for advice and have a nice day;D

[pcslinfo]

-{ Quote: "I don't think PEID is a good tool to judge validity of PE files (or any other, for that matter - other than Windows loader itself); some files reported by PEID as invalid can be started without any problems, and vice versa." }-
Hi I_G
First many thanks for your pro advice.
PEID is a rough way to filter the non-PE file(txt,wsf,etc) out of the samples base. To actually or precisely judge the sample malicious or not, functional or not, we run each of them with exe extension and delete the samples who is clean, corrupted and not executable by guarding their behavior through hips in vmware.

PEID is a good way to reduce some non-sense workload and hope I have given you a good answer:)

Have a nice day
Jeff

[risl]

If you have 3000 samples, and if it would take 15 minutes to check one sample: It would be 45.000 minutes, which is 750 hours and that would be 12,5 days if you work 24hours per day. If you work 8 hours per day only on checking samples if they are malicious or not(which I doubt), it would be ~94 days which is over 3 months in time just starting and analyzing samples.

If we take account that there are holidays, weekends, bathroom breaks, coffee breaks, phone ringing, real/actual work, hobbies and stuff. It would be a massive job to analyze all the samples, even if there are multiple persons doing this.

[pcslinfo]

-{ Quote: "If you have 3000 samples, and if it would take 15 minutes to check one sample: It would be 45.000 minutes, which is 750 hours and that would be 12,5 days if you work 24hours per day. If you work 8 hours per day only on checking samples if they are malicious or not(which I doubt), it would be ~94 days which is over 3 months in time just starting and analyzing samples.

If we take account that there are holidays, weekends, bathroom breaks, coffee breaks, phone ringing, real/actual work, hobbies and stuff. It would be a massive job to analyze all the samples, even if there are multiple persons doing this." }-
First, thank you for your consideration.
As for us, only I need is to judge the sample is malicious or not, unlike the anti virus vendors, we need not to extract the signature and provide a anti virus solution. That is like, only I need to know is that the water is salty, but I needn't know what concentration it has. So 15mins is the time maybe for analyst in a AV vendor, not for us.

And for me, I have some colleagues to help me and we are used to process the sample selection.

To hold such selection process is nesseray to ensure the quality of the testing sample set. If the sample set is not good(eg. have some clean or corrupted ones), it will obviously affect the final result.

Once again, many thanks and have a good day;D
Regards
Jeff

[Baz_kasp]

-{ Quote: "We need some kind of agreement that AV vendors offically take part in our testing and we will not test them before that.
To test a product without any agreement will sometimes cause some problems such as copyright, etc.

Receive means I don't make virus and they are all coming through different channels. And can you tell me what is the trustworthy way in your mind?

Thank you very much for advice and have a nice day;D" }-


I wouldn't say "untrustworthy"....but what is to stop your "sources"...whatever/whoever they may be from distributing those said samples to vendors before you perform the testing?

[trjam]

Baz, that could be said about all testing sites. In the end there is a trust factor but there is also one sure fire way of knowing. Looking at all testing sites collectively. I mean if certain vendors are always ranked near the top, then it it a safe bet that is reality. In the end it doesnt matter who those top 5 are because all will just about equally keep you safe.

[Baz_kasp]

-{ Quote: "Baz, that could be said about all testing sites. In the end there is a trust factor but there is also one sure fire way of knowing. Looking at all testing sites collectively. I mean if certain vendors are always ranked near the top, then it it a safe bet that is reality. In the end it doesnt matter who those top 5 are because all will just about equally keep you safe." }-


I'm not being bitter (omg avira got 100% etc) or anything like that, just making an observation ;D

It would be great if Jefrey could clarify what those sources are.... honeypots, private collectors etc

[pcslinfo]

-{ Quote: "I wouldn't say "untrustworthy"....but what is to stop your "sources"...whatever/whoever they may be from distributing those said samples to vendors before you perform the testing?" }-

e.g.
Some weblink malware, like http://www.?????.com/ss11.exe, we have several ways to collect such kind of malware. But as it exsits in the real internet, AV vendors will also have ways to get them e.g. cloud security ways. I can only ensure that I will not provide the testing samples before I finished the testing.

While in the other hand, I will provide the AV vendors(no matter they join the testing or not) after I finished the testing to let them verify the result.

Regards
Jeff

[pcslinfo]

-{ Quote: "I'm not being bitter (omg avira got 100% etc) or anything like that, just making an observation ;D

It would be great if Jefrey could clarify what those sources are.... honeypots, private collectors etc" }-

I have some auto-generated web-malware download list and the sample will auto updated in 1 second. That is to say, one day, I can at least download several thousand of such malware. Almost all of them are executable and malicious. And several samples are new to many security vendors. But, do you think I should add them to testing bed? In almost of time, normal users will never be infected by them. So we only add prevalent samples and these samples are collected one month before each time of testing.

In the mamual, I have point out the channels to get the samples:
Independent security research organizations, famous security forums and live web link malware. It is not a secret;D

Always welcome your observations and this will help a lot to improve our testing:) Thanks a lot!
Jeff