New test from Anti-Malware.ru [Archive]

C.S.J

January 9th, 2009, 11:10 AM

impressive from drweb once again, shame it was 4.44 tested, but nevermind :)

defensewall as everyone knew, achieved high scores, that thing is a beast, just a little annoying to run. lol

take away the paranoid detectors and AV's that include hips modules, and its AVG and bitdefender who achieve great results with drweb just behind ;)

eset? :thumbd: :wacko:

.. but then again, its just another one of those tests :blink:

the more i see them, the more im left un-impressed and to be honest, un-interested lately. :-\

Thankful

January 9th, 2009, 11:32 AM

If this analysis uses Virustotal, it has already been discredited:
http://www.prevx.com/blog/106/Why-using-VirusTotal-for-AV-testing-is-a-bad-idea.html
Accurate AV testing should be left to professionals such as AV-Comparatives, AV-Test.org, etc.

virtumonde

January 9th, 2009, 11:35 AM

-{ Quote: "impressive from drweb once again, shame it was 4.44 tested, but nevermind :)

defensewall as everyone knew, achieved high scores, that thing is a beast, just a little annoying to run. lol

take away the paranoid detectors and AV's that include hips modules, and its AVG and bitdefender who achieve great results with drweb just behind ;)

eset? :thumbd: :wacko:

.. but then again, its just another one of those tests :blink:

the more i see them, the more im left un-impressed and to be honest, un-interested lately. :-\" }-

You forgot your favourite Avira ;D who has for now only the guard which is still very good
Bitdefender has a behaviour blocker and a registry control.I don't know if those components ware used and what was considered as pass if so as i didn't understand the methodology exactly.

funkydude

January 9th, 2009, 11:36 AM

I really would like another "professional" company to do tests, not this virustotal/honeypot/corrupt files nonsense.

Thankful

January 9th, 2009, 11:38 AM

-{ Quote: "I really would like another "professional" company to do tests, not this virustotal/honeypot/corrupt files nonsense." }-
I agree 100%.

C.S.J

January 9th, 2009, 11:40 AM

since when do AM use VirusTotal, have i missed something? :blink:

TonyW

January 9th, 2009, 11:43 AM

Also most people won't come into contact with the test set files or similar unless they're going into areas where they exist.

C.S.J

January 9th, 2009, 11:44 AM

-{ Quote: "You forgot your favourite Avira ;D who has for now only the guard which is still very good
Bitdefender has a behaviour blocker and a registry control.I don't know if those components ware used and what was considered as pass if so as i didn't understand the methodology exactly." }-
lol, totally expected, just read between the lines...

"Avira Premium Security has been effective because of the high level of detection of exploits (see Table 3 of the full record of the test), and packed objects (specifically, the detection of malicious software used in the Packer)."

it does not offer good protection, regardless of what anyone on this forum may say. :wacko:

also, i forgot about BitDefenders behaviour controls, so AVG it is then :thumb:

even more impressed with the good olde doctors result :)

Thankful

January 9th, 2009, 11:44 AM

From the article:

-{ Quote: "The selection of Malware
This means that the links to downloadable samples of malicious software should not were detected by antivirus file more than 20% from the list of tested products, which tested through the service VirusTotal (all at the service connected to 38 different antivirus engines).. If selected samply and detected by someone who is usually the verdicts were inaccurate (suspected infection or wrapped object)." }-

Zombini

January 9th, 2009, 11:44 AM

They tested NIS2008 instead of NIS2009. The differences between SONAR on those two versions is night and day. Thats why NIS did so poorly with 08.. Why in the world do these "tests" compare 08 and 09 products of different vendors.

C.S.J

January 9th, 2009, 11:46 AM

-{ Quote: "They tested NIS2008 instead of NIS2009. The differences between SONAR on those two versions is night and day. Thats why NIS did so poorly with 08" }-
hmm maybe, but it was up against drweb 4.44 which is an old engine now.

Einsturzende

January 9th, 2009, 12:03 PM

From 34 samples Kaspersky failed just 1, for 3 there is only partially infection (main infection is avoided and system files and core of OS are intact), rest it passed, I am really glad to see someone done on execution test, This is real and complete degree test. BRAVO thanks and keep it up :thumb:
BTW where is Comodo
P.S. From those kind of tests people can clearly see that Matousec tests (or at least scoring methodology) do not worth too much, look at Outpost

Arup

January 9th, 2009, 12:18 PM

Avira did superb as usual with its high level of detection, it was but to be expected.

bellgamin

January 9th, 2009, 12:59 PM

-{ Quote: "If this analysis uses Virustotal, it has already been discredited:
http://www.prevx.com/blog/106/Why-using-VirusTotal-for-AV-testing-is-a-bad-idea.html
Accurate AV testing should be left to professionals such as AV-Comparatives, AV-Test.org, etc." }-Prevx found fault with a test that doesn't show them in the best light. That's hardly an objective opinion. On the other hand, I notice that those apps which do well do NOT find fault.

*If you can't raise the bridge, lower the water.* :dry:

Thankful

January 9th, 2009, 01:32 PM

-{ Quote: "Prevx found fault with a test that doesn't show them in the best light. That's hardly an objective opinion. On the other hand, I notice that those apps which do well do NOT find fault.

*If you can't raise the bridge, lower the water.* :dry:" }-
If you had bothered to read the article, you would see that often an
AV's techniques cannot be implemented by an online scanner.
It has nothing to do with Prevx.

Killtek

January 9th, 2009, 01:32 PM

Oh.. so they tested NIS2008. Why would they use NIS2008 when 2009 has been out for three months???

Murack

January 9th, 2009, 01:42 PM

-{ Quote: "From the article:" }-

in which part of it is? where it is written?

thanks :)

ps: excellent reporting

Regards

Murack

January 9th, 2009, 01:44 PM

-{ Quote: "Oh.. so they tested NIS2008. Why would they use NIS2008 when 2009 has been out for three months???" }-

Hi,

perhaps because the test was done in October 2008

Regards :)

Thankful

January 9th, 2009, 01:44 PM

-{ Quote: "in which part of it is? where it is written?

thanks :)

ps: excellent reporting

Regards" }-
Here:
http://www.wilderssecurity.com/showpost.php?p=1382711&postcount=13

Murack

January 9th, 2009, 01:47 PM

-{ Quote: "Here:
http://www.wilderssecurity.com/showpost.php?p=1382711&postcount=13" }-

yes, I have already seen

but I do not find this part of the text

where?
In the "Methodology used in testing antiviruses for the treatment of active infections"?
or "Analysis of active infections treatment test results and awards"?
or in report pdf?

thanks :)

Thankful

January 9th, 2009, 01:50 PM

-{ Quote: "yes, I have already seen

but I do not find this part of the text

where?
In the "Methodology used in testing antiviruses for the treatment of active infections"?
or "Analysis of active infections treatment test results and awards"?
or in report pdf?

thanks :)" }-
Методология сравнительного тестирования The methodology of comparative testing

Murack

January 9th, 2009, 02:03 PM

-{ Quote: "Методология сравнительного тестирования The methodology of comparative testing" }-
here?
http://www.anti-malware-test.com/?q=node/54

sorry, but I do not see it... :(

Thankful

January 9th, 2009, 02:17 PM

-{ Quote: "here?
http://www.anti-malware-test.com/?q=node/54

sorry, but I do not see it... :(" }-
Here:
http://www.wilderssecurity.com/showpost.php?p=1382577&postcount=1

Murack

January 9th, 2009, 02:41 PM

-{ Quote: "Here:
http://www.wilderssecurity.com/showpost.php?p=1382577&postcount=1" }-

ah...ok :)

but I do not understand the reason for the difference between the two pages, articles

:blink:

Baz_kasp

January 9th, 2009, 04:13 PM

It's a nice insight...notice that all AV's were used on default settings so if you were to tweak it a bit and notch up the settings (e.g. interactive as opposed to automatic HIPS mode, extra detection categories) then they would probably have scored higher.

fce

January 9th, 2009, 05:21 PM

let me first congratulate the winner.....congratz KIS! :thumb:

subset

January 9th, 2009, 05:55 PM

??? ...congratz DefenseWall :thumb:

LoneWolf

January 9th, 2009, 06:27 PM

-{ Quote: "let me first congratulate the winner.....congratz KIS! :thumb:" }-

-{ Quote: "??? ...congratz DefenseWall :thumb:" }-

Yeah, I'd have to say the winner of that test was indeed DefenseWall.

Baz_kasp

January 9th, 2009, 06:40 PM

-{ Quote: "Yeah, I'd have to say the winner of that test was indeed DefenseWall." }-

Defensewall isn't an AV...or a suite... plus it was noted on the article:

-{ Quote: "Products and Safe'n'Sec DefenceWall HIPS differ in the way of interaction with users. If Safe'n'Sec work on the principle similar to the anti-virus products, and requires no special training, in relation to DefenceWall is not so simple. To learn how to effectively use the latter must be at least have some knowledge and experience, and carefully read the user manual." }- ...as opposed to automatic/semi automatic handling of malware compared with traditional av products or suites.

I think that is what they meant when they said KIS is the winner.

subset

January 9th, 2009, 10:24 PM

@ Baz_kasp

Is the interactive mode of KIS (like you advised in post #29 for higher score) easier to handle than DW?

Cheers

Baz_kasp

January 10th, 2009, 05:32 AM

-{ Quote: "@ Baz_kasp

Is the interactive mode of KIS (like you advised in post #29 for higher score) easier to handle than DW?

Cheers" }-

No idea. Haven't used DW!

LoneWolf

January 10th, 2009, 06:12 AM

-{ Quote: "Defensewall isn't an AV...or a suite" }-

This is correct, but it was included in the test and scored higher then the others.

Also this is why I use DW, because it is not an AV or a Suite.
DW's protection surpasses any av out there.
But one must use what one is comfortable with.

format_c

January 10th, 2009, 06:14 AM

-{ Quote: "let me first congratulate the winner.....congratz KIS! :thumb:" }-

ROTFL. did you expect the another winner?

anti-malware.ru belongs to the KasperskyLabs employee Iliya Shabanov used to learn an astrophysics but working in marketting division of KL. and there is one brigand there from an labourers' block of big city. kewl situation ;)

GES/POR

January 10th, 2009, 06:14 AM

-{ Quote: "It's a nice insight...notice that all AV's were used on default settings so if you were to tweak it a bit and notch up the settings (e.g. interactive as opposed to automatic HIPS mode, extra detection categories) then they would probably have scored higher." }-

This is the way to go, testing on default settings and vendors will crank up their defaults n make damn well sure they dont affect resources usage us much as now and also increase compatibility. Novice and expert users will benefit much from this.

Arup

January 10th, 2009, 06:39 AM

dawgg

January 10th, 2009, 07:34 AM

Good to see a test detecting threats really in the wild and driveby downloads, seeing as thats how many infections come along.

Thankful and funkydude, read the text properly, Virustotal was NOT used for the end results to see which AV detects it, it was ONLY used to initially find the samples and render it "new" enough to be used in the test or not.
-{ Quote: "-{ Quote: "I really would like another "professional" company to do tests, not this virustotal/honeypot/corrupt files nonsense." }-I agree 100%." }-Again, read the text properly, sounds to me you two are jumping to conclusions simply by reading the name "Virustotal" and trying to discredit the test without knowing whats going on. Re: Virustotal - read what I wrote before I quoted you
Re: honeypot - whats that got to do with this? honeypots weren't used
re: corrupt files - if they were corrupted, DW wouldn't have blocked 33/34 infections. If anything, that just suggests only 1 sample *may* have been corrupted... but hey, it *may* also be that DW may have simply not prevented it.

-{ Quote: "take away the paranoid detectors and AV's that include hips modules, and its AVG and bitdefender who achieve great results with drweb just behind ;)" }-
Why would one take away the AVs which include HIPS - at the end of the day, protection of the AV matters, that's its job, regardless of what method they use to protect users.
(.... and AVs need usability, lightness, low FP etc, we all know the drill ;D )

Kees1958

January 10th, 2009, 07:38 AM

I have played a lot with DW, and also have played with the new Avira 9 beta. It's advanced heuristics are very good, I estimated that the new Avira will have a 85% detection ration of zero day malware.

See http://www.wilderssecurity.com/showpost.php?p=1375098&postcount=61

When DW with outbound protection arrives and Avira 9 is out (free version), I will test them again, because it is the safest and most silent combo around! (currently runing the safest and fastest combo possible = Malware Defender plus Avast free)

Cheers Kees

Baz_kasp

January 10th, 2009, 07:47 AM

-{ Quote: "ROTFL. did you expect the another winner?

anti-malware.ru belongs to the KasperskyLabs employee Iliya Shabanov used to learn an astrophysics but working in marketting division of KL. and there is one brigand there from an labourers' block of big city. kewl situation ;)" }-

FUD alert ::)

http://www.anti-malware-test.com/?q=node/42

I don't see how they could have doctored the results either..... they used real malware samples, most of which can still be downloaded to this moment. KIS blocked the most out of the suites/av's. I dont see why this is such a problem.

C.S.J

January 10th, 2009, 07:53 AM

-{ Quote: "
Why would one take away the AVs which include HIPS - at the end of the day, protection of the AV matters, that's its job, regardless of what method they use to protect users.
(.... and AVs need usability, lightness, low FP etc, we all know the drill ;D )" }-
because its my choice to.

if i looked at it your way, DefenseWall would be the winner, every day, every month, every year..... no need for no tests. :wacko:

everyone looks at the tests different, ie. some people just look at the figures, everyone is different.

Tu5

January 10th, 2009, 08:40 AM

-{ Quote: "

When DW with outbound protection arrives and Avira 9 is out (free version), I will test them again, because it is the safest and most silent combo around! (currently runing the safest and fastest combo possible = Malware Defender plus Avast free)

Cheers Kees" }-

Im using avira 8 and defensewall 2.45 on one of my machines right now and it is IMO also one of the fastest and most effective combos possible. Not too bothered about defensewall with outbound protection as i already have LnS which does the job fantastically. But i will definatly be purchasing the new avira 9 8) .

format_c

January 10th, 2009, 10:09 AM

-{ Quote: "FUD alert ::)
http://www.anti-malware-test.com/?q=node/42
" }-

a scram of the well known lier.

-{ Quote: "
I don't see how they could have doctored the results either..... they used real malware samples, most of which can still be downloaded to this moment. KIS blocked the most out of the suites/av's. I dont see why this is such a problem." }-

I can create the set of sample which give me any result I want. even zero detection of KL/other products.

if you want to know - we kindly asked mr. Shabanov to don't test any of ours products, we don't trust him at all. but he did ignore all of our requests. now, we're considering him as unequal and mendacious person.

"must die, must die, this Jesus must, Jesus must, Jesus must die" (c) ;D

Thankful

January 10th, 2009, 11:56 AM

-{ Quote: "Good to see a test detecting threats really in the wild and driveby downloads, seeing as thats how many infections come along.

Thankful and funkydude, read the text properly, Virustotal was NOT used for the end results to see which AV detects it, it was ONLY used to initially find the samples and render it "new" enough to be used in the test or not.
Again, read the text properly, sounds to me you two are jumping to conclusions simply by reading the name "Virustotal" and trying to discredit the test without knowing whats going on. Re: Virustotal - read what I wrote before I quoted you
Re: honeypot - whats that got to do with this? honeypots weren't used
re: corrupt files - if they were corrupted, DW wouldn't have blocked 33/34 infections. If anything, that just suggests only 1 sample *may* have been corrupted... but hey, it *may* also be that DW may have simply not prevented it.

Why would one take away the AVs which include HIPS - at the end of the day, protection of the AV matters, that's its job, regardless of what method they use to protect users.
(.... and AVs need usability, lightness, low FP etc, we all know the drill ;D )" }-
Thank you for proving our point. 1. Do you feel comfortable having Virustotal or the author determining what is actual malware? 2. From the translated article, "It is important to note that all anti-virus were tested with the standard default settings and with all relevant updates, obtained in an automatic mode." Can't you see that default settings for some AVs are not the strongest?

This is why it is important to have professional organizations such as AV-Comparatives, AT-test.org, VB100, etc. do AV testing. Very few people have the appropriate knowledge and discipline to do accurate testing.

Zombini

January 10th, 2009, 01:09 PM

Zombini

January 10th, 2009, 01:10 PM

-{ Quote: "FUD alert ::)

http://www.anti-malware-test.com/?q=node/42

I don't see how they could have doctored the results either..... they used real malware samples, most of which can still be downloaded to this moment. KIS blocked the most out of the suites/av's. I dont see why this is such a problem." }-

They tested KIS2009 against NIS2008. That is an apples to oranges comparison. KIS2008 would have done a lot worse.

Dark_Hanzo

January 10th, 2009, 01:58 PM

Even if they tested NIS2009 instead of NIS2008, I don't think the detection rate will miraculously jump from 12% to, let's say, 70% or 80%.

dawgg

January 10th, 2009, 02:38 PM

-{ Quote: "because its my choice to.

if i looked at it your way, DefenseWall would be the winner, every day, every month, every year..... no need for no tests. :wacko:

everyone looks at the tests different, ie. some people just look at the figures, everyone is different." }-
I know everyone looks at them different, was just asking why... and now I know why :)

dawgg

January 10th, 2009, 02:59 PM

-{ Quote: "Thank you for proving our point.
1. Do you feel comfortable having Virustotal or the author determining what is actual malware?
2. From the translated article, "It is important to note that all anti-virus were tested with the standard default settings and with all relevant updates, obtained in an automatic mode."
Can't you see that default settings for some AVs are not the strongest?

This is why it is important to have professional organizations such as AV-Comparatives, AT-test.org, VB100, etc. do AV testing. Very few people have the appropriate knowledge and discipline to do accurate testing." }-
1. If the author didn't determine whats actual malware, who would? Who do you think verifies the "professional organization's" samples? What makes you think AM's testers do not have the appropriate knowledge and "discipline" to do accurate testing?
2. I'm not disputing that and testing using default settings do not mean the test is flawed. Using default settings wont show the maximum potential of the detection/prevention of the AV, but its the optimum setting the AV vendor thinks will have the best protection/performance/usability for most of its customers. Surely if the default protects/detects better, it implies the AV protects better for most "average" users?
All users don't use AVs at advanced settings, although those who do would know their AV *might* do better than the results of the test if they are using more secure settings.
This test is using default settings and I personally see no need to complain about that although sure, some of us may be better protected than the test results show because we may not all be using default settings.

The main problem with this test I can see is the the lack of samples, but doing tests where each sample is executed is generally a problem with these types of tests due to lack of resources

Thankful

January 10th, 2009, 04:11 PM

-{ Quote: "1. If the author didn't determine whats actual malware, who would? Who do you think verifies the "professional organization's" samples? What makes you think AM's testers do not have the appropriate knowledge and "discipline" to do accurate testing?
2. I'm not disputing that and testing using default settings do not mean the test is flawed. Using default settings wont show the maximum potential of the detection/prevention of the AV, but its the optimum setting the AV vendor thinks will have the best protection/performance/usability for most of its customers. Surely if the default protects/detects better, it implies the AV protects better for most "average" users?
All users don't use AVs at advanced settings, although those who do would know their AV *might* do better than the results of the test if they are using more secure settings.
This test is using default settings and I personally see no need to complain about that although sure, some of us may be better protected than the test results show because we may not all be using default settings.

The main problem with this test I can see is the the lack of samples, but doing tests where each sample is executed is generally a problem with these types of tests due to lack of resources" }-

From the article :"To test selected links to sites affected only the latest examples of malicious software. What does it mean «newest»? This means that the links to downloadable samples of malicious software should not were detected by antivirus file more than 20% from the list of tested products, which tested through the service VirusTotal (all at the service connected to 38 different antivirus engines). If selected samply and detected by someone who is usually the verdicts were inaccurate (suspected infection or wrapped object)."

1. Only the newest samples, ACCORDING TO VIRUSTOTAL, were included in the sample. I.e. only if less than 20% of the companies, ACCORDING TO VIRUSTOTAL, caught the samples, would they be included in the samples.

2. Default settings have a much greater effect on those companies that depend more on heuristic detection than on signatures.

After looking at the excel file, it seems to verify whether samples are actual malware based on the classification of Avira and on the classification of Kaspersky. What if Avira and/or Kaspersky are wrong? Is this any way to conduct a test?

fce

January 10th, 2009, 04:26 PM

why lots of people can't accept KIS top this test?

better luck next time to NIS and Avira fanboy ;D

dawgg

January 10th, 2009, 04:30 PM

-{ Quote: "1. Only the newest samples, ACCORDING TO VIRUSTOTAL, were included in the sample. I.e. only if less than 20% of the companies, ACCORDING TO VIRUSTOTAL, caught the samples, would they be included in the samples.

2. Default settings have a much greater effect on those companies that depend more on heuristic detection than on signatures." }-
1. The malware tester found samples IN THE WILD, on websites (as shown on the link to table 3 in AM's article), checked if its malicious, if it was malicious, checked how many AVs detected it and then kept it if less than 20 AVs detected it. Dont know how you're thinking they done it, but VT was not used to gather the samples or check if its malicious or not. VT was only used to ensure it was a fairly recent sample... no point testing a sample if 100% of AVs will detect it, it would be a waist of time.

2. Yes, and partially the AVs which use other technologies such as HIPS and maybe default update frequency

Zombini

January 10th, 2009, 06:53 PM

-{ Quote: "why lots of people can't accept KIS top this test?

better luck next time to NIS and Avira fanboy ;D" }-

Just like KIS wouldn't do so well if you tested the older KIS2008, same thing for NIS2008.. you have to test KIS2009 against NIS2009. Then we shall see who tops this test.

The results as they are mean nothing to me.

Zombini

January 10th, 2009, 06:55 PM

-{ Quote: "Even if they tested NIS2009 instead of NIS2008, I don't think the detection rate will miraculously jump from 12% to, let's say, 70% or 80%." }-

NIS2008 has no HIPS like heuristics.

NIS2009 has very effective HIPS-like heuristics. Makes all the difference in the world on this test since it is non-signature based test

BrendanK.

January 10th, 2009, 08:08 PM

Please also remember that this is prevention, not detection :shifty:

TrojanHunter

January 11th, 2009, 07:29 AM

Not surprised really with Kaspersky because it consistently does well in various testing, but nothing detects 100%. I just wish they would fix those slow updates and I've found Kaspersky slows down Internet browsing speed more so than other solutions I have tested.

If only Kaspersky could follow Nortons example of how to create fast software.

Bunkhouse Buck

January 11th, 2009, 07:42 AM

-{ Quote: "NIS2008 has no HIPS like heuristics.

NIS2009 has very effective HIPS-like heuristics. Makes all the difference in the world on this test since it is non-signature based test" }-

I agree with you. The entire heuristic matrix has been enhanced, so a test based on anything but the latest version cannot be valid- at least for NIS 2009.

Bunkhouse Buck

January 11th, 2009, 07:43 AM

-{ Quote: "Please also remember that this is prevention, not detection :shifty:" }-

Yes but it had to detect something before it could be prevented.

mnosteele

January 11th, 2009, 11:34 AM

Why would you guys even discuss a test with 15 samples? There are hundreds of thousands pieces of malware out there, 15 samples is a COMPLETE joke of a test.

::)

C.S.J

January 11th, 2009, 01:24 PM

-{ Quote: "Why would you guys even discuss a test with 15 samples? There are hundreds of thousands pieces of malware out there, 15 samples is a COMPLETE joke of a test.

::)" }-
you mean 34

aigle

January 11th, 2009, 05:06 PM

-{ Quote: "Why would you guys even discuss a test with 15 samples? There are hundreds of thousands pieces of malware out there, 15 samples is a COMPLETE joke of a test.

::)" }-
Test is still important as these were live smples from live sites, not a load of malware that you might never come across. It sure gives u a rough idea IMO.

format_c

January 12th, 2009, 03:29 AM

-{ Quote: "you mean 34" }-

LOL, Chris! 34, yeah-yeah! :)

TechOutsider

January 12th, 2009, 06:38 PM

-{ Quote: "NIS2008 has no HIPS like heuristics.

NIS2009 has very effective HIPS-like heuristics. Makes all the difference in the world on this test since it is non-signature based test" }-

NIS08 does intergrate "SONAR". Otherwise known as a form of HIPS.

Kees1958

January 13th, 2009, 02:37 AM

-{ Quote: "Test is still important as these were live smples from live sites, not a load of malware that you might never come across. It sure gives u a rough idea IMO." }-

I agree,

Also for Avira it is an impressive result, because

A) In the wild examples extracted from honey pots tend to have a regional influence, so this could favour companies with a lot of users or information sources in that region.

B) Tester also used known bulletins, so this more or less localises to the known on-line/web world (with a lower representation of Chinese/Asian virusses)

With this in mind Kapersky (besided being a top notch AV product) would have user base advantage (besides kown on-line world also strong representation in the East and specifically also in China)

dawgg

January 13th, 2009, 12:04 PM

-{ Quote: "Test is still important as these were live smples from live sites, not a load of malware that you might never come across. It sure gives u a rough idea IMO." }-Although there is allways the argument of the method of getting the samples and whether the author "selected" the samples or not... thats personal oppinion and experience if you want the answer of that.

Kees1958

January 13th, 2009, 04:26 PM

-{ Quote: "Although there is allways the argument of the method of getting the samples and whether the author "selected" the samples or not... thats personal oppinion and experience if you want the answer of that." }-

Indeed, but . . .

I can not recall where, but somewhere I have read that on averag 6 new malware samples per day were found in 2008. SO they had 34 samples or nearly six days sampling. When this was collected in two months (for ease of argument), then the six day collection would still represent 10% of the total. When having numbers above 1000 in more or less homogeneous domains, then, statistical relevance often is achieved with samples > 5 percent. With groups of around 100, you often need sample sizes of 10 to 15 percent (also depending on the variation of the researched population).

Now I have no idea, but everyone tells me that most malwares are variations of same families, so the test with 34 samples for zero-day testing could well be large enough to give us a fair impression (meaning statistical relevant). Another reason to belive why this is a nice test, is that AV-comparatives also has scores of between 20% - 75% for retroperspective tests. The scores did not vary any more than those of AV with huge numbers, so I am inclined to put some value in this test also.

Cheers

Kees