Malware-Test Lab: Antivirus Comparison Report (February 26, 2007)

i dont think its nice of people to slag off the person doing these tests, not nice at all.

also, he is trying to improve it, so why not give him some advice to improve it eh? its always nice to have test results of some kind, and if he is asking and willing to improve it, why not help eh?

also i dont think f-secure is bloated, maybe just on older machines now, the vista compat ones work well with it, ive noticed no bloat when using the beta vista.

 

He's basically asking IBK to sort out the mistakes in their test for them using IBK's resources. Testers with any credibility should do this themselves..

Londonbeat

 

Well, while that is correct to an extent, and most testers have the required tools, it is somewhat difficult to get the right tools for sorting the samples if you do not have the right sources. No need to say any more, but a certain tester has been trying to improve his sample set and has got absolutely no help from anyone, not even some advice on where to start....(No, not Malware-Test).

 

I just would like to remind that av-comparatives tests also undergone some criticism in alt.comp.anti-virus in the beginning (see e.g. http://groups.google.com/group/alt.comp.anti-virus/browse_frm/thread/6d81d8b81a9b8aac ). So we'd better try to help malware-test to improve their test set and methodology. Here is my contribution.

Bad samples can be sorted in 3 categories:
- non working files
- non malware files
- duplicates

Below are some hints on how to spot potentially bad samples. Those should be flagged/sorted and investigated individually (e.g. by running it under VMWare). You should also keep a record of the status of each sample known good (i.e. tested)/known bad/not tested/duplicate.

Removing broken files:
------------------------

- Truncated samples:
Some samples captured by the honeypots are truncated. I think it can sometimes be detected by checking the PE file structure, which can be automated.

- Badly repacked samples:
Some packers or some combinations of packers tend to produce non-functionnal samples. You'll have to test it yourself to find out which combinations work and which do not work... However, I believe that this kind of samples are more likely to be found in collections provided by services virustotal or jotti.

- Damaged samples:
Samples flagged as "damaged" by some AV scanners*, that may or may not be able to run or to perform malicious actions, and should be tested individually.

- Non replicating (buggy) parasitic viruses:
Parasitic viruses are (or can be) one of the most sophisticated kind of malware. And they are quite often buggy. Since you'll probably not have enough time to replicate every sample properly, the better solution might be to simply exclude those from your test set.


Removing non-malware files:
-------------------------------

- Unwanted programs:
Some samples might be categorized as "unwanted programs" or "vitools"*. It can be portscanners, trojan "editors" or "client" parts, ftp servers, etc. These should IMHO not be included in the test set. I never tried it, but I think that a simple way to flag files that *may* belong to this category would be to observe imported functions. When there is a lot of functions that have something to do with GUI (analysis of imports can be automated), it might mean that you are facing such a virtool.However, you should keep in mind that there are many Delphi malware that are so badly written that they import tons of functions they never use...

- Dropped/downloaded components:
If a sample has been added to the collection just because it was dropped (or downloaded) by a known malware, it should be checked carefully. Although it is often malware, there are malware that drop legitimate components.

- Executables packed with an "exotic" packer:
Some antiviruses (who said QuickHeal ?) are known to flag every and all files that have been processed by an "exotic" packer. These detections are usually simple to spot, and should always be verified by hand.

- Other "standard" false positives:
I do not know any "cheap" tecnique to spot these.

Removing duplicates:
-----------------------

- Polymorphic viruses:
Be careful not to add hundreds of samples of the same polymorphic virus to your collection! Different sha1/CRC mean nothing. Unless you really know what you are doing, you should avoid polymorphic viruses in your test set.

- Backdoors/dialers:
There are plenty of backdoors, downloader or dialer samples that differ only by e.g. the name of the IRC or web server to connect to, or the phone number to call. But it should (in my opinion) be consdered as one single sample. These differences are not always obvious if the maware is packed. Malwares with similar file size and/or detection names* should be checked. There are also droppers that only differ by the file they drop.

- Worms:
There are worms that append random data to themselves just to avoid detection by CRC/file size. There are also worms that transmit themselves inside password protected archives (with changing passwords). Again, similarity of detection names can be a hint. Automatic comparison of the first hundred bytes of the files is also easy to implement.


sai7sai, if you manage to remove most of the bad samples corresponding to these descriptions and to get sure that all your files are not included in archives and have the good extension, then I think your collection will be better than most.

* of course, relying on the output of the programs that you are supposed to test is far from an ideal methodology. However, pragmatically, it can help to perform a first sort.

 

That's for the tests.
Maybe in the real world they makes the difference.

 

Thanks for your suggestions.

 

http://www.people.frisk-software.com/~bontchev/papers/virlib.html