Snakeoil or not

[Blackcat]

A recent thread has discussed and then flamed the antivirus tests carried out by commercial magazines. I would like to add to this the results of a very recent test in the UK and also to widen the discussion on where we can obtain reliable information on the capabilities of AV Scanners.


A relatively new magazine in the UK- PC Extreme- has just published an interesting comparison of 24 Antivirus scanners and 6 trojan cleaners. This was interesting for a number of reasons;

1. The number of programs tested - invariably in the UK, in magazine testing the number of scanners has been between 4 and 10 and with one exception, Norton or McAfee have received the Editors Choice. So no surprises there!!!! Some of the scanners used in the test I had never heard of - Fire and Solo for example.

2. This test was solely on detection ability of the scanners and their scan speed and not other criteria.

3. NOD32 was tested. The first time I have seen this carried out outside of the virus bulletin.


Each on demand scanner was tested against 47,000 'viruses' including file, Dos, macro, Windows and script viruses, malware and trojans. 'All antivirus programs had the latest engine upgrades, updates and the scan was made with maximum heuristics set in all programs' ( 5-10th November, 2002).

The overall detection rate was as follow;the top 10 being;

1. F-Secure->99%; 2. KAV4->99%; 3.McAfee-97%; 4.RAV-95%; 5. E-Scan-95%' 6.F-Prot-92%; 7. PC-Cillin-92%; 8. Sophos-90%;9. Norton-89%; 10.Dr Web- 88%.


The results for only antitrojan detection were very similar to the above with Avast replacing Dr Web in the top 10.
In contrast, AVG, E-Trust, Ikarus, VirusBuster and Quick Heal brought up the rear in both categories. So overall these results are what we would have generally expected.

However the big shock was with NOD32( under the name NodIce32) which was listed only 19th in overall detection and 14th in antitrojan detection. NOD for example only picked up 59% of the trojans tested. NOD was not the only AV program to 'underperform'; Norman and Panda also were well away from the top 10 listed above.

From the above I would like to make the following comments;

1. Why was NOD the only one of accepted 'excellent' AV Scanners ( KAV, RAV, Dr Web, E-Scan) to underperform? This was not only in trojan detection but also in virus detection e.g. only 76% of 2703 script viruses picked up. Whereas the relatively poor scanners e.g. Ikarus and VirusBuster were in there perceived positions i.e. bringing up the rear. Is this commercial test therefore judged to be 'rubbish' because NOD was not number 1?

NOD has conquered all in the virus bulletin results over the last few years but it is difficult to judge the overall effectiveness of a scanner when you can pass the test for a particular month but not pick up all the viruses and also fail the test by catching all the viruses but producing too many false positives. I am not stating that NOD has missed any viruses here but apparently some other 'Passed' scanners have. I am not flaming NOD as I am a registered owner of this program( together with KAV4 and Dr Web)but I am somewhat puzzled that this scanner seemed to be the only one that seriously underperformed. In addition from what I can see this new magazine does not appear to be as commercial as some others in the UK which only carry recommendations for Symantec products.

2. Commercial sites such as Cnet are now I think well known to support only the big buck programs such as Norton and most people do not take their own reviews seriously, particularly after the recent review of NOD. Although the reviews of customers I do find interesting and more truthful. Therefore judge these sites with caution.

3. Forums here and elsewhere are also informative and I have switched to NOD and Dr Web with information from here. So I have found this forum very useful for choosing security programs although there has been some comments of late that wilders shows a bias towards NOD and shoots down those which are not followers( not my words).

Therefore where do people suggest we go for unbiased, reliable information about the performance of AV Scanners?

[Firefighter]

To Blackcat from Firefighter!

The results from that magazine seems to be almost identical that were in the Technodrome24 site made on May 2002.

So the measurements may be quite reliable?

Regards,
Firefighter!

[rodzilla]

> Each on demand scanner was tested against 47,000 'viruses' including file, Dos, macro, Windows and script viruses, malware and trojans. 'All antivirus programs had the latest engine upgrades, updates and the scan was made with maximum heuristics set in all programs' (5-10th November, 2002).

Who was the tester ?

Does he have credentials and credibility in the antivirus world ... or is he another unknown self-appointed "virus expert" like CNet's Ken Feinstein and the myriad no-name wannabes who post their "test results" in alt.comp.virus ?

How many of those 47,000 "viruses" did the tester individually execute on a clean system ?

Was each of those 47,000 "viruses" individually infection-validated, or did the tester simply download a bunch of VX collections and take someone else's word that they were viruses ?

Are all 47,000 of those "viruses" guaranteed to be real, live viruses ?

Did PC Extreme, like CNet, use "simulated" viruses ?

No reputable antivirus vendor would give an "outsider" even one live virus, let alone 47,000 ... so where did PC Extreme's tester get them ?

> However the big shock was with NOD32 (under the name NodIce32)

Calling an antivirus program by a name which hasn't been used for nearly five years shows a great deal of intelligence, wouldn't you say ... or did the tester actually use a five-year-old NOD-ICE in his test ? :)

> which was listed only 19th in overall detection

ROFL

This alone shows the test is BS!

NOD32 has not missed one single solitary "in the wild" virus in a Virus Bulletin test since May 1988 and has numerous clean sweeps of every virus in every category from VB's "zoo" under its belt.

I find it extremely difficult to believe that a PC magazine journalist (or anyone else, for that matter) would have a more comprehensive, more up-to-date, 100% validated virus suite than Virus Bulletin's ... and impossible to believe that anyone in the world would have a suite of validated viruses which would put NOD32 at #19 in detection.

> and 14th in antitrojan detection. NOD for example only picked up 59% of the trojans tested.

Although it detects a large number of the more common Trojans, NOD32 does not claim to be a Trojan detector, so this part of the test is invalid. (Would you test TDS or Tauscan against viruses ?)

> 1. Why was NOD the only one of accepted 'excellent' AV Scanners ( KAV, RAV, Dr Web, E-Scan) to underperform?

By design, NOD32 deliberately ignores non-infected files which many other scanners tag as infected.

Part of Mele20's original beef with Eset stemmed from the fact that several (five, from memory) scanners identified a file in her collection as being infected with Magistr, while NOD32 "missed" it.

The file wasn't infected ... so NOD32 ignored it.

A few similar "detections" of non-viruses by other scanners in PC Extreme's 47,000 collection would leave NOD32 way behind ... but which scanner would you say did the better job ?

> Is this commercial test therefore judged to be 'rubbish' because NOD was not number 1?

VX collections are invariably filled with crud ... broken viruses, corrupted viruses, non-viruses, etc ... in fact, in more than 15 years in the antivirus industry, I have not seen one online virus collection which didn't contain crud.

Unless every single one of the 47,000 "viruses" used in the test was 100% tested, infection-validated, and guaranteed to be a live, infectious virus (as is every single virus used by Virus Bulletin) then the test would be rubbish even if NOD32 had won.

> NOD has conquered all in the virus bulletin results over the last few years but it is difficult to judge the overall effectiveness of a scanner when you can pass the test for a particular month but not pick up all the viruses and also fail the test by catching all the viruses but producing too many false positives.

I repeat ... NOD32 has not missed a single "in the wild" virus in a Virus Bulletin test since May 1998. Nothing else in the world comes even close to this detection figure.

But ... I know what you mean. In the November 2000 VB100 test, NOD32 was the only product in the world to make a clean sweep of 100% of every virus in every category, but it missed the award because of a false positive. (Norton AntiVirus missed 299 viruses, but still won the award.)

> although there has been some comments of late that wilders shows a bias towards NOD and shoots down those which are not followers(not my words).

If you check through the forums you'll find the only people who have been "shot down" over NOD32 are those whose complaints have been proved wrong. (DSL is widely regarded as a "Norton shill site". You can't please everyone.)

> Therefore where do people suggest we go for unbiased, reliable information about the performance of AV Scanners?

The short answer is "Virus Bulletin".

I haven't always agreed with Virus Bulletin, and I've had a few fights with them over the past 14 years (see http://www.nod32.com.au/nod32/awards/vb0207.htm for info on a couple of them) but I have always regarded VB as the world's #1 independent antivirus product tester ... even when I was distributing Kaspersky Antivirus and VB kept putting NOD32 out in front in detection. :)

[Firefighter]

To Rodzilla from Firefighter!

I don't distrust the measurements that the magazine did, because they were too near the Technodrome24 site test to be pure casual.

It is totally different task to scan 100% proof test among less than 10 000 viruses, from which only some 500 are in the Wild like VB 100% tests do.

I believe that test was typical in the Zoo test, which is not within VB:n tests.

Do you really believe that NOD had found such an unique invention among 30..50 AV-developers, that is capable to scan and detect all ever made viruses so incredible fast?

Why that kind of intelligency really keeps in one producer hands for a long time as it seems to be now?

Regards,
Firefighter!

[root]

Hi Blackcat. I'm probably somewhat like you in that I look for various tests that seem to give some meaningful information about the quality of the products I use.
I also have tried most AVs at one time or another and perform my own tests from time to time on my own computer. Not professional testing, but it lets me know what AVs pick up the live viruses, packed and unpacked on my machine with my configuration.
Interesting that F-Secure did so well in that test, because on my machine, the last time I checked it in detecting a mere 15 or so recent viruses, it only caught 10 or 11. At the same time, AVP 3.5 caught all. Curious, since F-Secure uses KAVs engine. I ditched F-Secure because of that.
I am now using DrWeb as my resident protection, and so far, it hasn't missed anything I have thrown at it.
I think Rod brought up many good points. One of my first impressions about the test you mentioned, was that 47,000 is a lot of viruses. I wonder too at the quality of those samples, and how many of them are really old and no longer in circulation. AVs do drop very old definitions that no longer pose a threat.
An AVs performance in tests depends on two things for the most part. First the virus definitions need to be complete and up to date. Second, the engine has to have the ability to detect viruses in their various states and mutated forms.
Oversimplification, maybe, but take KAV. It has an excellent detection engine and unpacker. It has a huge and usually up to date definition base. So, when it misses even one virus in a test, one of two things has happened. Either that virus or its mutation is not in the virus definitions, or it is packed in a way that KAV cannot unpack it to check it.
At least, that is the impression I get. I am talking about tests only. Looking at it in that light, gives me the impression that all testing is therefore of some value, but still limited in determining how well a given AV will perform in the real world.
I guess what I'm trying to say is that for me, no test in the world is going to to be more than another piece of information to add to the pile.
If a person were to take that test and use the results to pick the top three candidates for their use, the DrWeb and NOD32 would not even be considered. That, to me would be a huge mistake.
Also, if a person were to use the test results at Virus Bulletin, you might well throw out KAV for consideration. Bad move again in my opinion.
So, for me, I just keep looking and testing, and try to have fun surfing the net. ;D

[rodzilla]

> To Rodzilla from Firefighter!

> I don't distrust the measurements that the magazine did, because they were too near the Technodrome24 site test to be pure casual.

You can believe whatever you like ... or you can believe the world's #1 independent antivirus tester. (Some people still think the world is flat.)

> It is totally different task to scan 100% proof test among less than 10 000 viruses, from which only some 500 are in the Wild like VB 100% tests do.

Rubbish! Do you have any idea what Virus Bulletin tests ?

> I believe that test was typical in the Zoo test, which is not within VB:n tests.

More rubbish! Find me the "zoo" that has no crud! "Zoo" tests are and always have been worthless!

> Do you really believe that NOD had found such an unique invention among 30..50 AV-developers, that is capable to scan and detect all ever made viruses so incredible fast?

No antivirus program can detect all viruses ever made.

> Why that kind of intelligency really keeps in one producer hands for a long time as it seems to be now?

Perhaps because NOD32 proves itself to be a better virus detector when tested by real antivirus experts ?

According to CNet, Norton AntiVirus has been the best virus detector in the world in every test since 1996. Do you believe that too ?

[Firefighter]

To Rodzilla from Firefighter!


Calm, calm, when I was at school over 35 years ago my religion's teacher said that no one gets 10, the best performer may have 9/10, the 10 belongs to God.

What I am trying to say is that in every test made we must overburn every system far over it's limits. Only then we can get some knowledge of the capabilities that system has.

So, when a test winner has 100% result, the test is not made perfectly.

I think if the winner makes 50...100 mistakes, then it has cleared up the limits of every system.

It's an other story, if in normal life there will be that kind of situations, but who knows?

I am using DrWeb as my backup despite of those test results, so what! ::)

"The truth is out there"

Regards,
Firefighter!

[Technodrome]

VirusP's test that I got on my site is more or less rubbish. Why?

Because of this:

1. Use of Binary viruses, BeOS, FreeBSD, Linux, OS2, Unix, BinaryImage, BAS, HLL*.* etc.
Test was done on Windows 98 machine!

2. 20887 out of 43843 were MS-DOS viruses

3. 8065 out of 43843 were DoS, Constructors, Exploit, Flooders, Hoax, Jokes, Nukers, Sniffers, Spammers, Virus Tools, Corrupted, Droppers, Intended, PolyEngines.


Technodrome

[Firefighter]

To Rodzilla from Firefighter again!

June-2002 VB test: On Demand test

AVG missed 115 ITW viruses, which were 13.09 % of ITW viruses, so ITW total were about 879 viruses.

Quickheal missed 181 macro viruses, which were 4.55 % of macro viruses, so macro total were about 3 978 viruses.

AVG missed 410 polymorphic viruses, which were 16,25 % of polymorphics, so the total were about 2 523 viruses.

Hauri missed 628 standard viruses, which were 32.45 % of standards, so the total standards were about 1 935.

Thus the total summary of tested viruses were then about
9 315 viruses concerning to the On Demand test.

If that is not true, then the VB June 2002 acrobat file is not correct.


What happens when there will be the 9 316. virus in real life? Is that not a potential risk at all?

In quality world there is a limit called six sigma, and outside that the risk that system collapses is minimal, but it is still one measure among others. I don’t know the six sigma levels concerning the amount of viruses to be detected, that you are in safe with six sigma risks. But I think the majority of all AV-developers don’t know it either, because there is so huge fluctuation among virusbases. It is very difficult to measure, because some virus may be over 1000 times common than the other one.

When I wrote some 10 000 viruses to be detected in VB tests, it was based that VB June acrobat file, no more or less!


“The truth is out there”


Regards,
Firefighter!

[rodzilla]

-{ Quote: " quoting: Technodrome link=board=24;threadid=6463;start=0#43204 date=1042998962]
VirusP's test that I got on my site is more or less rubbish. Why?

Because of this:

1. Use of Binary viruses, BeOS, FreeBSD, Linux, OS2, Unix, BinaryImage, BAS, HLL*.* etc.
Test was done on Windows 98 machine!

2. 20887 out of 43843 were MS-DOS viruses

3. 8065 out of 43843 were DoS, Constructors, Exploit, Flooders, Hoax, Jokes, Nukers, Sniffers, Spammers, Virus Tools, Corrupted, Droppers, Intended, PolyEngines.

Technodrome

" }-


I rest my case! :) :)

[rodzilla]

> So, when a test winner has 100% result, the test is not made perfectly.

So Virus Bulletin has been repeating the same mistake for five years ? :)

[rodzilla]

-{ Quote: " quoting: Firefighter link=board=24;threadid=6463;start=0#43231 date=1043005241]
To Rodzilla from Firefighter again!

June-2002 VB test: On Demand test

AVG missed 115 ITW viruses, which were 13.09 % of ITW viruses, so ITW total were about 879 viruses.

Quickheal missed 181 macro viruses, which were 4.55 % of macro viruses, so macro total were about 3 978 viruses.

AVG missed 410 polymorphic viruses, which were 16,25 % of polymorphics, so the total were about 2 523 viruses.

Hauri missed 628 standard viruses, which were 32.45 % of standards, so the total standards were about 1 935.

Thus the total summary of tested viruses were then about
9 315 viruses concerning to the On Demand test.

If that is not true, then the VB June 2002 acrobat file is not correct.


What happens when there will be the 9 316. virus in real life? Is that not a potential risk at all?

In quality world there is a limit called six sigma, and outside that the risk that system collapses is minimal, but it is still one measure among others. I don’t know the six sigma levels concerning the amount of viruses to be detected, that you are in safe with six sigma risks. But I think the majority of all AV-developers don’t know it either, because there is so huge fluctuation among virusbases. It is very difficult to measure, because some virus may be over 1000 times common than the other one.

When I wrote some 10 000 viruses to be detected in VB tests, it was based that VB June acrobat file, no more or less!


“The truth is out there”

Regards,
Firefighter!
" }-


Virus Bulletin has not been testing against the same set of viruses since 1988. :)

[rodzilla]

> The results from that magazine seems to be almost identical that were in the Technodrome24 site made on May 2002.

> So the measurements may be quite reliable?

Not according to Technodrome.

[rodzilla]

Does anyone have any idea how long it would take to individually test and properly infection-verify 47,000 viruses ?

Assume you have a clean W98 system containing all the goat files you need to test every known type of file and macro virus.

Attempted infection, verification of the infection, and re-installing the clean OS and goat files would take what ... ten minutes per virus ?

In round figures, that equates to one man working 24 hours a day, 7 days a week, for more than a year, without a break.

If you worked 8 hours a day, 5 days a week, with 1.5 hours off for breaks, it would take you more than 4.5 years to validate your 47,000 viruses.

If you put a four-man team on the job, working 8 hours a day, 5 days a week, with 1.5 hours off for breaks, you could have it completed in just over one year.

I doubt that PC Extreme paid four guys a year's salary apiece to do nothing but validate 47,000 viruses. :)

(That's the short version, btw. It doesn't take into account that some viruses infect only DOS files, some infect only NT/2000/XP files, some infect only the MBR or bootstrap, some are multipartite, some are polymorphic, etc ... and your "10 minutes per virus" average increases enormously because you have to test all the "failed to infect on W98" samples again on other operating systems.)

As Firefighter said, "The truth is out there" . . . . . . but you won't find it in PC Extreme!

[Technodrome]

-{ Quote: " quoting: rodzilla link=board=24;threadid=6463;start=0#43296 date=1043027045]
As Firefighter said, "The truth is out there" . . . . . . but you won't find it in PC Extreme!
" }-

LOL ;D



Technodrome

[Straight Shooter]

Well then, just how did PC Extreme do it?!?!?!?!?

Well, then, if what you're saying is true... (which I know it is, what you're saying, because it makes sense) that means that their are snake oil reports aimed at basically fooling "jubs" like me and half of the other newbies here at Wilders to go "gaga" over an AV which may be good, but not as "wonderful" as the magazine suggests...

I mean, let's face it, people out there are trying to knock out NOD32's position...C/NET made a VERY lame attempt, and now it's PC Extreme's turn..

And besides, correct me if I'm wrong, but why should I be worried about zoo viruses? From what I know, they haven't been publicly released.. They are in a virul lab, locked away somewhere...If my av can detect some of them heuristically, so much the better.. But why worry about them?

Why worry about a disease that doesn't exist.?

This is getting to the "anti virul Hypochondria" stage... LOL

Remember that "jpg" virus that someone created, then sent a copy to McAfee? (A couple of months ago)

Everyone with av software has a def for a virus that only the av labs and the author has.. I never heard of it being released to the public, unless I'm wrong...

I don't think these magazines really test ALL these virures.. I think they plaguarize a little from here, a alittle from there.. From the new av reviews I read at C/NET, I think C/NET doesn't do anymore wide scale testing.. I think you scared them off, Rod! (LOL)

[rodzilla]

> Well then, just how did PC Extreme do it?!?!?!?!?

With smoke and mirrors ? :) :)

At last count (some 18 months or so ago) there were more than three million individual "virus" files online for download in various VX collections. Most were duplicates, of course ... but I doubt that there are more than 20 x 47,000+ virus suites in existence in the world which are 100% validated and verified ... and there is no way the owners of those suites would ever give them to anyone outside the antivirus industry.

> Well, then, if what you're saying is true... (which I know it is, what you're saying, because it makes sense) that means that their are snake oil reports aimed at basically fooling "jubs" like me and half of the other newbies here at Wilders to go "gaga" over an AV which may be good, but not as "wonderful" as the magazine suggests...

The world is filled with self-appointed "virus experts" ... there's one on every streetcorner. Mix these in with the swarm of shills and marketroids and spin doctors and hype artists and scare mongers and doom-and-gloomers who actually work for antivirus vendors and you'll be forgiven for thinking "The antivirus industry thrives on its own snake oil." (Rob Rosenberger's http://www.vmyths.com is a good read if you want to see some of the tall tales spread by the antivirus industry over the years.)

> I mean, let's face it, people out there are trying to knock out NOD32's position...C/NET made a VERY lame attempt, and now it's PC Extreme's turn..

Life's tough at the top. :)

In 1992 I went into partnership with ESaSS, the creator of the (then) virtually unknown Thunderbyte. Thunderbyte became rated as the world's #1 virus detector ... and self-appointed "virus experts" worldwide tried to knock it down.

In 1995 I went into partnership with Kaspersky Lab, the creator of the (then) virtually unknown AVP. AVP became rated as the world's #1 virus detector ... and self-appointed "virus experts" worldwide tried to knock it down.

When I went into partnership with Eset, NOD32 was virtually unknown. NOD32 is now rated as the world's #1 virus detector ... and self-appointed "virus experts" worldwide are trying to knock it down.

Deja vu is a part of my life! :)

> And besides, correct me if I'm wrong, but why should I be worried about zoo viruses? From what I know, they haven't been publicly released.. They are in a virul lab, locked away somewhere...If my av can detect some of them heuristically, so much the better.. But why worry about them?

Many "zoo" viruses have never been seen outside laboratories. Over the years I've been given many viruses by their authors (on a "not to travel" basis) which have never seen the light of day. They're not included in VX "zoos" because the authors didn't trust their fellow VXers to not distribute them ... and they're not included in antivirus vendors' or Virus Bulletin's "zoos" because I'd given my word that I wouldn't give them to anyone. (Some "hobbyist" virus coders have never released a virus.)

The other side of the coin, as pointed out by Technodrome, is that most "outside the antivirus industry" collections contain so much crud and non-virus junk that the "test results" are worthless in the real world.

A recent classic example of "crud detection" creating false impressions was Mele20's corrupted Magistr file which NOD32 "failed to detect". Had that file been included in a Virus Bulletin VB100 test, the scanners which tagged it as "infected" would have been disqualified.

That's just one crud file!

Imagine the false impression created by dozens/hundreds of them!

> Why worry about a disease that doesn't exist.?

Precisely!

> This is getting to the "anti virul Hypochondria" stage... LOL

Wear condoms on all ten of your fingers as you type! :)

> Remember that "jpg" virus that someone created, then sent a copy to McAfee? (A couple of months ago)

Yep. Snake oil to the max!

> Everyone with av software has a def for a virus that only the av labs and the author has.. I never heard of it being released to the public, unless I'm wrong...

Usenet virus newsgroups are regularly visited by some VX wannabe who claims to have written (or collected) 50 viruses which nothing in the world can detect. Somehow we manage to survive the attacks from these script kiddies.

> I don't think these magazines really test ALL these virures.. I think they plaguarize a little from here, a alittle from there..

Did CNet's Ken Feinstein really test NOD32 ? Try to duplicate his findings! You can't! It's impossible!

The sheer logistics of the 100% validation of 47,000 viruses put it beyond the reach of most computer magazines ... and beyond the reach of most newbie antivirus company startups.

A one-man-band wanting to break into the antivirus industry these days has a lot of work ahead of him to establish his verified virus suite, because no ethical antivirus man will give him live samples. If he started working on his 47,000 collection today, he might have it completed by 2010. :)

> From the new av reviews I read at C/NET, I think C/NET doesn't do anymore wide scale testing.. I think you scared them off, Rod! (LOL)

ROFL

I've heard that rumor too. :)

[Tinribs]

I think people get a little too caught up in tests and test results that appear all over the net by so called experts.
Nod32 has performed very well for, its light on resources and unobtrusive, it hasn't let me down yet, thats all the tests I need so far.
If the day ever comes when it misses a 'common' well known virus and lets me down then maybe then its time to look elsewhere, but so far it hasn't and thats good enough for me. ;)

[rodzilla]

> I think people get a little too caught up in tests and test results that appear all over the net by so called experts.

Yep ... they're everywhere. :)

> Nod32 has performed very well for, its light on resources and unobtrusive, it hasn't let me down yet, thats all the tests I need so far.

Good thinking!

> If the day ever comes when it misses a 'common' well known virus and lets me down then maybe then its time to look elsewhere, but so far it hasn't and thats good enough for me.

When a new "undetectable by anything" virus appears, someone has to be first to go down. You'd be extremely unlucky to be that person. However, NOD32's list of "detected on sight" viruses is very impressive ... CIH, Marburg, Melissa, LoveLetter, Anna Kournikova, Homepage, and many more "big name" viruses were nailed by NOD32's heuristics "before they were written". (This is why we don't need to release an update every five minutes.) :)

Anyone who tells you his antivirus program can detect 100% of viruses 100% of the time is a liar ... but the history of reputable independent professional tests shows that NOD32 has been consistently closer to 100% than anything else for the past five years ... and we intend to keep that record intact.

There's no "perfect" antivirus solution ... but NOD32 coupled with common sense is as close as you can get.

[Firefighter]

To Rodzilla from Firefighter about so called "VB statistics"


Hi, you mentioned somenthing about "real" viruses. Let's take a break for a while and look a bit backwards.

I counted from VB statistics the amount of tested viruses September-1998, September-2000, June 2002 and at last August-2002, like I did it before. No one has said, that the viruses to be tested were the same all these years, so stupid they hopefully were not.

All these years 1998-2002 for example, what I have observed among my closest neighbourhood (some 15 households), the number of home PC:s have increased from 2 to 9, so it was more than quadrupled during these 4 years. Only two households have been chanced during these years, and the new ones have now no PC:s at all. I can't be thinking, that the development in the rest of world has not been something very different.

So let's think that there are 3-4 times more home PC users now, that it was 1998, it is only a rough estimate, but that is not the point. So the amount of potential virus developers have also strongly increased. So the amount of new detected viruses has heavily increased all the time.

All these years the IT-world has been struggling with enormous economical difficulties. Hundreds of thousands employees have been fired. Very huge amount of capable virus developers have a lot of time to do something about.

After all that happened, so what has happened to for example VB tests.

Septemper 1998, WinNT:

ITW viruses of tested: 753
Macro viruses of tested: 1 510
Polymor. vir of tested: 14 244
Standard vir of tested: 1 015

Total amount of tested September 1998 : 17 522

September 2000, NetWare:

ITW viruses of tested: 1 069
Macro viruses of tested: 4 048
Polymor. vir of tested: 3 261
Standard vir of tested: 1 614

Total amount of tested September 2000 : 9 992

June 2002 winXP:

ITW viruses of tested: 879
Macro viruses of tested: 3 978
Polymor. vir of tested: 2 523
Standard vir of tested: 1 935

Total amount of tested June 2002: 9 315

August 2002 NetWare:

ITW viruses of tested: 2 000
Macro viruses of tested: 4 250
Polymor. vir of tested: 1 656
Standard vir of tested: 2 500

Total amount of tested August 2002 : 10 406

So there were any marks of that, the amounts of measuments were increasing, despite of that the total amount of real viruses were strongly increasing all the time. Real living viruses are never a fixed number, it increases all the time so far the virusdevelopers exists and home PC users number increases.

If I said before that they don't even know the amount of viruses to be tested, I may be wrong. They can't do so many tests it is needed, because they haven't enough money to make it. It seems to be more or less AV-producers facelift, that those have been controlled by independent factors like VB, that about something have been done to this thing. In pure statistical rules of games that has nothing to do it.

And at last about so called statistics, here are some figures of VB August 2002 tables:

ITW viruses missed:

Mcafee: 1 ; missed 0,04 %; so total ITW was 2 500
VirusBuster: 1 ; missed 0,05 %; so total ITW was 2 000

Macro viruses missed:

VET Antiv. 16 ; missed 0,29 %; so total Macro was 5 517
DrWeb: 34 ; missed 0,8 %; so total Macro was 4 250
Sophos 9 ; missed 0,23 % ; so total Macro was 3 913

Polymorphic.viruses missed:

Norman 149 ; missed 8,75 %; so total Polymorphic was 1 703
RAV 78 ; missed 4,71 % ; so total Polymorphic was 1 656
Sophos. 93 ; missed 6,69 %; so total Polymorphic was 1 390

Standard viruses missed:

Norman. 15 ; missed 0,68 %; so total Standard was 2 206
VirusBuster 11 ; missed 0,44 % ; so total Standard was 2 500
Sophos: 17 ; missed 0,57 %; so total Standard was 2 982

So I counted manually the total tested number of each argument: (amount of misses/ missed percents) = totally tested viruses in each argument.

From these figures it seems to me that they don't know the amount of tested viruses from each category at all. ITW varies 2 000-2 500, Macro 3 913-5 517, Polymorphic 1 390-1 703 and Standard 2 206- 2 982. That is unforgivable in making statistics, where only facts are them to be counted, no lotteries at all. Does no one believe them at all anymore, where is the truth?

We may be fools, but never so stupid that we couldn't even count in percents.

You might be more right than you thinked yourself. Making statistics is very costly, now even VB 100% seems to have difficulties like this, because they had to hire employees, who couldn't even count percents..

I never had respected authorities, before I have estimated their skills to do their own job. There are none on this earth to say that there is only one truth, one right statistics etc.

I think, we have not to be worried about what, or how , I am writing something, We should be worried about very strongly, do we really know at all what's happening.

I am not a spokesman to any AV-developer and I get on well among minorities, that's why for example my backup is DrWeb, resident is RAV and so on. But now it seems to be that I have to look for my happiness from minoritie's minority! :-[


"The truth is out there"

Regards,
Firefighter!

[VirusP]

It's the first time i am taking part in this forum, and that wouldn't happen if BlackCat hadn't informed me of the existance of this forum.

I am the webmaster of URL removed - TOS violation and also a well-known and highly respected virus collector for the last 5 years, so some of you may have seem me in several irc underground virus-related channels.

1) As for what FireFighter says, the tests we made by me, with no co-operation with anyone else, so the results are utterly true.

2)As for rodzilla, i really do not know my credentials in the antivirus world, but i do know some avers know me by my nick, since i have often tried to contact them...of course, noone answered :-X
In fact, i used to think highly of the aver community, until the day i found out all the nasty and ugly things goin on underground, inside the aver community. And i mean all the competition, the ignorance and the myth about "av know it all, vxers no zip". If only half of the world knew how many trades and help avers have got from some of us, you would be at least surprised...So, to conclude, i need no credentials from the avers, mate, i know where i stand, so do most of the "older" virus traders, and trust me, there are 1-2 virus traders more respected than many avers right now...

I did not have the luxury of testing 47,000 samples, nor most people do. Does this mean my tests are 100% false? I hope you do not think that av companies check thouroughly ALL of their samples, cause you'd also be mistaken. Besides, i am not the one that falsely names non-viral samples as a real virus ;)

And let me tell you this: i got the samples by searching over 4 years in the internet, so, theoratically, these are the same samples, more or less, that most people will get in their emails, so, the % of them being non-working is quite small. Afterall, do not forget that the fact that only one antivirus soiftware detects a sample doesn't make the sample false, but doesn't make the av program bad either; there is another option and it's called detection % and some av do not lack in it as much as most of the rest.

So, u think the test is BS?? Heheh, ok, you can think whatever you say, but the fact remains i can full your email up with 10,000 live virus samples and we'll see which one of us will laugh after you execute them...WANNA TRY IT ?????

It's not my fault NodIce can't detect my samples..maybe they do not use such a good identification in their program. But 11,000 samples are alot to be ALL non-viral, don't you think?....As for VBulletin, i do not trust completely, since almost every antivirus test is being paid-off these days, every "OFFICIAL" test at least.

Trojans..i suppose you don't think thousands of people suffer from trojans' infection, right? And i suppose they ain't the 2nd largest source of computer hacking, right?? So, when you go to the hospital, there should not be a cardiologist, but only the nurses right?? I mean, what does a cardiologist do in a hospital?...You need to change this attitude about trojans asap, imho.And so does nodice.

I get the feeling that you think i sabotaged nodice for some reason..if this is the case, lemme tell you, is this the BEST excuse you can find for the poor results of nodice??I surely think so. Money do not mean nothing to me, cause i ain't doin it for the $$$, just in case you haven't figured it out yet, so no reasons for me to sabotage nodice!!

3)Quote:

"VirusP's test that I got on my site is more or less rubbish. Why?

Because of this:
1. Use of Binary viruses, BeOS, FreeBSD, Linux, OS2, Unix, BinaryImage, BAS, HLL*.* etc.
Test was done on Windows 98 machine!
2. 20887 out of 43843 were MS-DOS viruses
3. 8065 out of 43843 were DoS, Constructors, Exploit, Flooders, Hoax, Jokes, Nukers, Sniffers, Spammers, Virus Tools, Corrupted, Droppers, Intended, PolyEngines.

Technodrome"

Dear Tehcnodrome, the fact that antivirus programs use the above descriptions in the database only shows the need for avers to make their vx dbases larger to be able to compete with each other..that ain't my fault as well >:(
And let me remind you, that it's exactly this descriptions that "help" ignorants decide which av to purchase. So, why should i keep them out of the test, since it's pretty obvious most antivirus programs will include them in the near future, if not already?

4) As for the avers in this forum:

And where exactly should the truth be searched for? Inside av related forums???
Join us in our channels and debate with 1-2 of us, we'll see whether you'd love to publish the logs from our discussion or not.

5) I currently have 99,000 virus samples, that means i still cannot be considered to be a serious virus collector?? ;D

6) I know all about nodice's attempt to get some good sales, which is goin on at this moment. Do not try to convince me otherwise, so maybe all this accusations against the credibility of the test's results is not that innosence. Easy to accuse someone of anything, isn't it??

7) Current detection of my 99,000 samples

34057 unique virii for AVP
31559 unique virii for F-Prot
19195 unique virii for DRWEB
16670 unique virii for MCAFEE
21339 unique virii for NOD
41348 unique virii for RAV

As for Nodice,

number of diagnosed files: 94510
number of viruses found: 68638
termination time: 18:29:00 total time: 653 sec (00:10:53)

What do you think?

8) I do not believe that my collection is 100% perfect. I do know how good av software are though, even if i hadnt't made these tests you so easily blame. This is where all avers have ALWAYS lacked in: the ability to open their ears and listen to the vxers. You should know already you are not virus gods either :-X

P.S. I would like to apologise for my often aggresive attitude, but it's hard to find out there is a forum, in which everyone says and accuses you of almost everything, especially when the accuser is the one that should be apologising for the performance of the antivirus program he suggests...I hope noone will take no offense in what i have posted.

Best regards

VirusP - VX trader

[Paul Wilders]

VirusP,

In no way I can - and feel th need to - reply to questions and remarks addressed to Rod and/or Eset.

Some overall remarks though:

-{ Quote: "[I hope you do not think that av companies check thouroughly ALL of their samples" }-

That's an overall statement; in case you do have proof in regard to specific av companies, you're welcome to post.

-{ Quote: "So, u think the test is BS?? Heheh, ok, you can think whatever you say, but the fact remains i can full your email up with 10,000 live virus samples" }-

While this is addressed to Rod: feel free to email me the URL from your database (much more convenient then fillinh up the inbox ;) ).

-{ Quote: "It's not my fault NodIce can't detect my samples..maybe they do not use such a good identification in their program. But 11,000 samples are alot to be ALL non-viral, don't you think?...." }-

See above. You probably agree the (viral) test bed is the issue here. Looking forward to verify your statement/test.

-{ Quote: "And where exactly should the truth be searched for? Inside av related forums???" }-

Is this a rethorical question? ;) You obviously don't trust virusbtn, The Uni from Magdenburg and others - you rely only on your own test bed and tests. That's just fine.Problem is (as far as I'm concerned) I can't verify, since you don't reveal the exact test bed, packagers used etc.

-{ Quote: "Trojans..i suppose you don't think thousands of people suffer from trojans' infection, right? And i suppose they ain't the 2nd largest source of computer hacking, right??" }-

IMHO trojans/backdoors do need a specific stand alone software to handle. AVs should handle ITW viruses first and foremost.

-{ Quote: "I currently have 99,000 virus samples, that means i still cannot be considered to be a serious virus collector??" }-

No problem here: you are a serious virus collector.

-{ Quote: "This is where all avers have ALWAYS lacked in: the ability to open their ears and listen to the vxers." }-

Sorry to hear that's your experience. It's opposite to our knowledge.

-{ Quote: "I hope noone will take no offense in what i have posted." }-

Please refrain from posting the URL from your vx site. It's against our rules.

regards.

paul

[Technodrome]

-{ Quote: " quoting: VirusP link=board=24;threadid=6463;start=15#52525 date=1048406743]
Dear Tehcnodrome, the fact that antivirus programs use the above descriptions in the database only shows the need for avers to make their vx dbases larger to be able to compete with each other..that ain't my fault as well >:(
And let me remind you, that it's exactly this descriptions that "help" ignorants decide which av to purchase. So, why should i keep them out of the test, since it's pretty obvious most antivirus programs will include them in the near future, if not already?

" }-

Dear VirusP

OT: Some antivirus products are able to detect virus by behavior, which requires execution of virii code. This is one of features that av product may offer. I don’t find many av testers that execute code and then test av product

I read you logs (I am reading them for almost 8-10 months). Even in your logs I see NOD32 strength in heuristic feature (this was the main reason for me to purchase NOD32 ). I’ve never purchased NOD32 because of VB 100 awards. I bought it because of its features. I agree that NOD32 could improve ZOO/Trojan virus detection and I hope they will. But to me “heuristic” engine is all what I need from this product.

In your test you included many non-virus related stuff! I don't see the reason for antivirus companies to detect these. Yes some AV will detect them but thats up to them.

I respect your testing and I M FOLLOWING it regularly. But I'll call it rubbish as long as you include non- virus related stuff.


Technodrome

[VirusP]

First of all, thank you Forum Admin for your partitipation. Allow me to answer most of your remarks, as made above:

1)I have no proof of my saying that av don't test all of their samples, i just happen to know it, maybe in some cases i am wrong indeed..
2)I had no intention of fillin' up Rod's email really :D, i just wanted to make a statement. My dbase is open to anyone for trading purposes only though ;)
3)I don't trust 100% the other av tests, exactly because they are made under many special sircumstances..i'd not like to refer any, i think you get my point.
4)Trojans may not be real viruses, according to the meaning of the term virus, still, most people don't spend money on both antivirus and anti-trojan software, they like a 2-in-1 solution, don't you think??
5)I really wish all avers would listen to our opinions.
6)I will delete the url from my profile since it is against your rules, np there.
7)Dear Technodrome, i absolutely agree with you in all issues, except for the last one. Since crap-ware ;) are really not virus samples, then why do more and more av programs include them in their dbases? Don't you think buyers will be affected by this? I happen to know some people that depend their av program choise by the number of the viruses referred inside the av dbase...Poor choises, by both people and av companies...
I also thank you for following my tests and i will be glad to hear any comments on them, besides, the first thing vxers know me by is my patience and will to debate in all vx issues.

P.S. I would like to mention that, ther main reasons for me using Nod for my dbase logs and trades, is the existence of a free dos version and the fact that it is a good and very promissing av program. Now, i don't know whether you will believe me, or not, but nodice was always one of my favs, despite of what i said earlier.

Thanks for tolerating me once more.

Best regards,

VirusP

[Technodrome]

-{ Quote: " quoting: VirusP link=board=24;threadid=6463;start=15#52552 date=1048442764]
7)Since crap-ware ;) are really not virus samples, then why do more and more av programs include them in their dbases? Don't you think buyers will be affected by this? I happen to know some people that depend their av program choise by the number of the viruses referred inside the av dbase...Poor choises, by both people and av companies..." }-
I could be a marketing plot but it could also be strength of AV Product (like KAV) to detect everything out there. :D Your test just confirmed that KAV is able to detect almost everything out there. (see http://www.av-test.com for similar tests). Some AV products claim to detect 80,000 + viruses (such as SOPHOS or Command) but in realty they COULD detect less then KAV with 64,000 +. It all depends on their virus signs counting system, or on their program structure. Every AV product has its advantage or disadvantage. There is nothing new about it.

-{ Quote: "I also thank you for following my tests and i will be glad to hear any comments on them, besides, the first thing vxers know me by is my patience and will to debate in all vx issues." }-

No Problems! Thank you for making this tests available. ;)
Naming test beds in future testing would be really helpful! 8)



Technodrome

[VirusP]

Naming test beds? What is that?

[Technodrome]

-{ Quote: " quoting: VirusP link=board=24;threadid=6463;start=15#52598 date=1048492197]
Naming test beds? What is that?
" }-

Virus names. ;)



Technodrome

[VirusP]

You mean keeping a log of the virus names of the samples used for the test, acxcording to each av?

[Technodrome]

Let's say you decided to test av products against 5,000 virii (zoo, itw, Polymorphic etc.), simple provide the names of 5,000 virii. ;)

PS logs won't tell us what virus is missed. ;)

Something like this http://www.checkvir.com/displayer.php?site=checkvir&page=2.3.10 . Look for "Used virus samples"


Technodrome

[VirusP]

-{ Quote: " quoting: Technodrome link=board=24;threadid=6463;start=15#52668 date=1048565477]
Let's say you decided to test av products against 5,000 virii (zoo, itw, Polymorphic etc.), simple provide the names of 5,000 virii. ;)

PS logs won't tell us what virus is missed. ;)

Something like this http://www.checkvir.com/displayer.php?site=checkvir&page=2.3.10 . Look for "Used virus samples"


Technodrome
" }-

You can download the list with the virus names of ALL antivirus programs used at my latest av test, from

http://www.virus.gr/avtestlist.rar

Looking forward to your comments this time ;)