New test from Anti-Malware.ru

1. If the author didn't determine whats actual malware, who would? Who do you think verifies the "professional organization's" samples? What makes you think AM's testers do not have the appropriate knowledge and "discipline" to do accurate testing?
2. I'm not disputing that and testing using default settings do not mean the test is flawed. Using default settings wont show the maximum potential of the detection/prevention of the AV, but its the optimum setting the AV vendor thinks will have the best protection/performance/usability for most of its customers. Surely if the default protects/detects better, it implies the AV protects better for most "average" users?
All users don't use AVs at advanced settings, although those who do would know their AV *might* do better than the results of the test if they are using more secure settings.
This test is using default settings and I personally see no need to complain about that although sure, some of us may be better protected than the test results show because we may not all be using default settings.

The main problem with this test I can see is the the lack of samples, but doing tests where each sample is executed is generally a problem with these types of tests due to lack of resources

 

From the article :"To test selected links to sites affected only the latest examples of malicious software. What does it mean «newest»? This means that the links to downloadable samples of malicious software should not were detected by antivirus file more than 20% from the list of tested products, which tested through the service VirusTotal (all at the service connected to 38 different antivirus engines). If selected samply and detected by someone who is usually the verdicts were inaccurate (suspected infection or wrapped object)."

1. Only the newest samples, ACCORDING TO VIRUSTOTAL, were included in the sample. I.e. only if less than 20% of the companies, ACCORDING TO VIRUSTOTAL, caught the samples, would they be included in the samples.

2. Default settings have a much greater effect on those companies that depend more on heuristic detection than on signatures.

After looking at the excel file, it seems to verify whether samples are actual malware based on the classification of Avira and on the classification of Kaspersky. What if Avira and/or Kaspersky are wrong? Is this any way to conduct a test?

 

why lots of people can't accept KIS top this test?

better luck next time to NIS and Avira fanboy

 

1. The malware tester found samples IN THE WILD, on websites (as shown on the link to table 3 in AM's article), checked if its malicious, if it was malicious, checked how many AVs detected it and then kept it if less than 20 AVs detected it. Dont know how you're thinking they done it, but VT was not used to gather the samples or check if its malicious or not. VT was only used to ensure it was a fairly recent sample... no point testing a sample if 100% of AVs will detect it, it would be a waist of time.

2. Yes, and partially the AVs which use other technologies such as HIPS and maybe default update frequency

 

Just like KIS wouldn't do so well if you tested the older KIS2008, same thing for NIS2008.. you have to test KIS2009 against NIS2009. Then we shall see who tops this test.

The results as they are mean nothing to me.

 

NIS2008 has no HIPS like heuristics.

NIS2009 has very effective HIPS-like heuristics. Makes all the difference in the world on this test since it is non-signature based test

 

Please also remember that this is prevention, not detection

 

Not surprised really with Kaspersky because it consistently does well in various testing, but nothing detects 100%. I just wish they would fix those slow updates and I've found Kaspersky slows down Internet browsing speed more so than other solutions I have tested.

If only Kaspersky could follow Nortons example of how to create fast software.

 

I agree with you. The entire heuristic matrix has been enhanced, so a test based on anything but the latest version cannot be valid- at least for NIS 2009.

 

Yes but it had to detect something before it could be prevented.

 

Why would you guys even discuss a test with 15 samples? There are hundreds of thousands pieces of malware out there, 15 samples is a COMPLETE joke of a test.

 

you mean 34

 

Test is still important as these were live smples from live sites, not a load of malware that you might never come across. It sure gives u a rough idea IMO.

 

LOL, Chris! 34, yeah-yeah!

 

NIS08 does intergrate "SONAR". Otherwise known as a form of HIPS.

 

I agree,

Also for Avira it is an impressive result, because

A) In the wild examples extracted from honey pots tend to have a regional influence, so this could favour companies with a lot of users or information sources in that region.

B) Tester also used known bulletins, so this more or less localises to the known on-line/web world (with a lower representation of Chinese/Asian virusses)

With this in mind Kapersky (besided being a top notch AV product) would have user base advantage (besides kown on-line world also strong representation in the East and specifically also in China)

 

Although there is allways the argument of the method of getting the samples and whether the author "selected" the samples or not... thats personal oppinion and experience if you want the answer of that.

 

Indeed, but . . .

I can not recall where, but somewhere I have read that on averag 6 new malware samples per day were found in 2008. SO they had 34 samples or nearly six days sampling. When this was collected in two months (for ease of argument), then the six day collection would still represent 10% of the total. When having numbers above 1000 in more or less homogeneous domains, then, statistical relevance often is achieved with samples > 5 percent. With groups of around 100, you often need sample sizes of 10 to 15 percent (also depending on the variation of the researched population).

Now I have no idea, but everyone tells me that most malwares are variations of same families, so the test with 34 samples for zero-day testing could well be large enough to give us a fair impression (meaning statistical relevant). Another reason to belive why this is a nice test, is that AV-comparatives also has scores of between 20% - 75% for retroperspective tests. The scores did not vary any more than those of AV with huge numbers, so I am inclined to put some value in this test also.

Cheers

Kees

 

Norton 2009 and 2008 blocks access to sites that attempt to exploit a vulnerability or drop malware.

I tried to execute two seperate installers of AV09; with NAV and Windows Firewall off; it "could not download installation file". Apprently my HOSTs file blocked access.

So, security must be layered. That test only tests one layer of the whole package. Informative; however a wider perspective must be needed to fully understand.