VirusP test in PC Utilities [Archive] - Wilders Security Forums

rodzilla

December 2nd, 2003, 03:07 AM

-{ Quote: " quoting: Zander link=board=24;threadid=17092;start=0#msg105679 date=1070276047]
It's got me stumped why a computer magazine would run a virus spreader's crappy AV tests, but one just did. The usual people are praising it on DSLReports Forums. http://www.dslreports.com/forum/remark,8620641
" }-

I read your comment to Vampirefo on DSL .........
=====
Vampirefo>> "Here is a challenge for you get your AV Company to offer $5.00 a piece for each virus they missed on VirusP's test. They only have to pay if EICAR says they are indeed viruses, no samples will be given to the AV company until money changes hands."

Zander> I've got a better idea, that will put an end to this argument, and if you're right, it won't cost you a cent. According to you, they are all viruses, so put your money where your mouth is and pay EICAR $5 each for every file in the collection that is *not* a virus.
=====

You shoot a good game of pool, Minnesota Fats! ;D

rodzilla

December 2nd, 2003, 03:22 AM

> Not all of them do praise this useless "test"

Most of the "in crowd" seem to think it's great.

> and as far as I know, the author is a virus collector, not a virus spreader

What's the difference ? "Collectors" (who are outside the antivirus industry) will trade viruses with anyone who comes along, with no regard for who they are or what they might do with the viruses.

> and certainly no tester.

Yep.

> Personally, I would recommend everyone to disregard such "tests".

I have little regard for any commercial computer magazine test apart from the few which utilize the resources of Virus Bulletin ... in my opinion they do a disservice to their readers and are a waste of the paper they're printed on ... and to pre-empt the predictable "You like Virus Bulletin because NOD32 is always #1 in its tests" wails from the peanut gallery, I'll point out that I'm on record as backing Virus Bulletin as the leading independent antivirus product tester all the way back to 1989.

Paul Wilders

December 2nd, 2003, 03:39 AM

Rod,

Most of the "in crowd" seem to think it's great.

No offense to them, but in my view "most of the crowd" never has been or will be a decive criterium for obvious reasons.

What's the difference ? "Collectors" (who are outside the antivirus industry) will trade viruses with anyone who comes along, with no regard for who they are or what they might do with the viruses.

Although I agree generally spoken this is true, I for one are giving the benefit of the doubt here, as long as there are no facts this person actually doe trade viruses.

I have little regard for any commercial computer magazine test apart from the few which utilize the resources of Virus Bulletin ... in my opinion they do a disservice to their readers and are a waste of the paper they're printed on ...

Harshly put - but indeed it's for good reasons all serious and major AV companies submit their product(s) themselves for VB testing.

regards,

paul

rodzilla

December 2nd, 2003, 03:55 AM

> And yes, this "test" is far from what it claims to be. I'd look elsewhere for my information if I were a reader of this magazine.

Right!

My opinion is, and has been for years, that only three antivirus product tests (and a few commercial computer magazine comparative reviews which utilize their resources) have any value in the real world. Virus Bulletin is #1 ... ICSA and Checkmark can fight it out for the #2 slot ... the rest have value only to tight little cliques of lemming-like sycophants who would worship Dark Avenger if he posted an antivirus product review.

(Roman (Rokop) is shaping up to be a serious contender ... and if he keeps improving his test bed and methodology then I might even include his tests in my "worthwhile" category sometime down the track.)

Having said that, there are a few (very few!) guys who I've known for years (Paul Wilders, Alex Byron, and Jan Wikstrom, are three who spring to mind) who know viruses and know antivirus software and know how to test it properly. I consider their tests valid and worthy of consideration (as a guide) because they put in the time and effort required to ensure that they test against 100% verified infectious files. Their tests are limited to some degree because their test beds are limited (in numbers) ... but they all test products against real live viruses which are out and about and are most likely to bite you at the time of the test ... not against VX "collections" containing hundreds/thousands of antique DOS viruses, boot sector images, lab samples, collector's one-offs, "simulated viruses", and unknown amounts of non-replicating crud.

[Soapbox mode OFF]

rodzilla

December 2nd, 2003, 04:10 AM

>> Most of the "in crowd" seem to think it's great.

> No offense to them, but in my view "most of the crowd" never has been or will be a decive criterium for obvious reasons.

Unfortunately, empty vessels make the loudest noise. :)

>> What's the difference ? "Collectors" (who are outside the antivirus industry) will trade viruses with anyone who comes along, with no regard for who they are or what they might do with the viruses.

> Although I agree generally spoken this is true, I for one are giving the benefit of the doubt here, as long as there are no facts this person actually doe trade viruses.

Do a Google search for "VirusP". The very first entry is "VirusP - VX Trading page - Virus collector".

"Trading" = "swapping" = "spreading" ... not "spreading" as in "spreading maliciously over the Internet" or whatever, but "spreading" nonetheless.

>> I have little regard for any commercial computer magazine test apart from the few which utilize the resources of Virus Bulletin ... in my opinion they do a disservice to their readers and are a waste of the paper they're printed on ...

> Harshly put

Harsh ... perhaps. Accurate ... definitely! :D

> but indeed it's for good reasons all serious and major AV companies submit their product(s) themselves for VB testing.

There is a small clique of "experts" on the security forums who ridicule Virus Bulletin at every opportunity, but they seem to forget that the VB100 is the award every antivirus vendor strives to win. (Virus Bulletin uses this as a slogan. They pinched it from me!) :)

Godzilla

December 2nd, 2003, 04:16 AM

-{ Quote: "

Unfortunately, empty vessels make the loudest noise. :)

" }-

;D ;D ;D

Paul Wilders

December 2nd, 2003, 04:29 AM

Rod,

Unfortunately, empty vessels make the loudest noise.

You should take the size of the vessel into account as well ;D

Do a Google search for "VirusP". The very first entry is "VirusP - VX Trading page - Virus collector".

"Trading" = "swapping" = "spreading" ... not "spreading" as in "spreading maliciously over the Internet" or whatever, but "spreading" nonetheless.

I stand corrected.

Harsh ... perhaps. Accurate ... definitely!

Grin...you've been known for your unique personal way of phrasing ;)

There is a small clique of "experts" on the security forums who ridicule Virus Bulletin at every opportunity

What else is new? I'm pretty sure their influence is hardly of any importance - especially in the big picture, and that's by no means security forums...

...but they seem to forget that the VB100 is the award every antivirus vendor strives to win.

Quite so, as I stated above.

(Virus Bulletin uses this as a slogan. They pinched it from me!)

Ever considered copyright? ;)

regards.

paul

rodzilla

December 2nd, 2003, 04:53 AM

>> Harsh ... perhaps. Accurate ... definitely!

> Grin...you've been known for your unique personal way of phrasing

I'm really a very modest guy. :) :) :)

>> There is a small clique of "experts" on the security forums who ridicule Virus Bulletin at every opportunity

> What else is new? I'm pretty sure their influence is hardly of any importance - especially in the big picture, and that's by no means security forums...

I get a laugh out of watching all the little dramas unfold. Every now and then a real IT security professional turns up on DSL, but they seldom stick around for long. Some German IT wizard made some convincing arguments against taking VirusP's previous test seriously some months ago, but he vanished after less than a week. Tetsu-ko would certainly know more about IT security (and viruses) than all the DSL regulars combined ... she's in a class of her own ... but the resident "experts" managed to piss her off in just a couple of days.

>> ...but they seem to forget that the VB100 is the award every antivirus vendor strives to win.

> Quite so, as I stated above.

Yep ... you did.

>> (Virus Bulletin uses this as a slogan. They pinched it from me!)

> Ever considered copyright?

I posted it all over the place, so I guess they figured it was Public Domain.

I don't mind ... it's good for my ego. :)

Paul Wilders

December 2nd, 2003, 05:14 AM

I'm really a very modest guy.

Sure you are ;D

I get a laugh out of watching all the little dramas unfold. Every now and then a real IT security professional turns up on DSL, but they seldom stick around for long. Some German IT wizard made some convincing arguments against taking VirusP's previous test seriously some months ago, but he vanished after less than a week. Tetsu-ko would certainly know more about IT security (and viruses) than all the DSL regulars combined ... she's in a class of her own ... but the resident "experts" managed to piss her off in just a couple of days.

Well, I'm (and that's a board policy) am not into bashing other boards/forums - and that goes for DSL as well. In all honesty, one can't blame Justin or WCB for the conduct from their audience/posters. In my opinion there are quite alot of people over there having lots of expertise on various issues. Justin and WCB have created a rather nice platform. It wouldn't be fair to judge them for the way some wish to post over there.

rodzilla

December 2nd, 2003, 09:38 AM

>> I get a laugh out of watching all the little dramas unfold. Every now and then a real IT security professional turns up on DSL, but they seldom stick around for long. Some German IT wizard made some convincing arguments against taking VirusP's previous test seriously some months ago, but he vanished after less than a week. Tetsu-ko would certainly know more about IT security (and viruses) than all the DSL regulars combined ... she's in a class of her own ... but the resident "experts" managed to piss her off in just a couple of days.

> Well, I'm (and that's a board policy) am not into bashing other boards/forums - and that goes for DSL as well.

Yep ... inter-forum wars are a pointless waste of time and bandwidth.

> In all honesty, one can't blame Justin or WCB for the conduct from their audience/posters.

I guess they try the best they can under trying conditions.

> In my opinion there are quite alot of people over there having lots of expertise on various issues.

Unfortunately the whispers of the people with real expertise are often drowned out by the shouts of the wannabes.

I just took a look at http://www.virus.gr/english/fullxml/default.asp?id=62&mnu=62

Obviously the True Believers haven't read it closely ... or they don't understand it.

VirusP> "The 58306 virus samples were chosen using VS2000 according to Kaspersky, F-Prot, RAV, Nod32, Dr.Web and McAfee antivirus programs. Each virus sample was unique by virus name, meaning that AT LEAST 1 antivirus program detected it as a new virus."

In plain English, what VirusP is saying is that if AT LEAST 1 of either Kaspersky or F-Prot or RAV or Nod32 or Dr.Web or McAfee tagged a virus as "live" then that was good enough verification for him.

Sorry ... it's nowhere near good enough for me ... and it's nowhere near good enough for the antivirus industry to recognize the test as valid.

VirusP> All "fake" virus samples were removed, as well as "garbage" files.

What methodology did VirusP use to determine which samples were "fake" viruses and which files were "garbage" ? Scan them with AT LEAST 1 of either Kaspersky or F-Prot or RAV or Nod32 or Dr.Web or McAfee ? ROFL

Disassembly will show with almost 100% accuracy that a given file is viral, but the only way to verify viral activity beyond all doubt is to execute the file on a susceptible operating system. If it's infectious then it's a virus. If it's not infectious then it's not a virus. No other methodology is acceptable!

VirusP> The virus samples were divided into these categories, according to the type of the virus :
[ . . . ]
VirusP> Malware = DoS, Constructors, Exploit, Flooders, Hoax, Jokes, Nukers, Sniffers, Spoofers, Virus Construction Tools, Virus Tools, Corrupted, Droppers, Intended, PolyEngines.

Wait a minute! Didn't VirusP just say All "fake" virus samples were removed, as well as "garbage" files. ?

If used in an antivirus test, what are Flooders, Hoaxes, Jokes, Nukers, Sniffers, Spoofers, Corrupted viruses, and Intended viruses if not "garbage" ?

They're certainly not the verified live viruses which must be used to the exclusion of everything else if a test of antivirus programs is to have any credibility!

Still, it could have been worse ... VirusP could have included a few hundred .ba$ files! :D

JimIT

December 2nd, 2003, 11:23 AM

-{ Quote: " quoting: rodzilla link=board=24;threadid=17092;start=0#msg106048 date=1070375939]
VirusP> All "fake" virus samples were removed, as well as "garbage" files.

VirusP> Malware = DoS, Constructors, Exploit, Flooders, Hoax, Jokes, Nukers, Sniffers, Spoofers, Virus Construction Tools, Virus Tools, Corrupted, Droppers, Intended, PolyEngines.

Wait a minute! Didn't VirusP just say All "fake" virus samples were removed, as well as "garbage" files. ?

If used in an antivirus test, what are Flooders, Hoaxes, Jokes, Nukers, Sniffers, Spoofers, Corrupted viruses, and Intended viruses if not "garbage" ?

They're certainly not the verified live viruses which must be used to the exclusion of everything else if a test of antivirus programs is to have any credibility!" }-

Well, I tried to point that out, to no avail. Doesn't matter which AV is one's "favorite"--the test is invalid for all of 'em.

...but, I was beating my head against the wall... :-\ ;) ;D

Paul Wilders

December 2nd, 2003, 11:36 AM

Rod,

In short: right on spot ;).

On a side note: seems like WCB (forum boss DSLR) as well as Kevin M. from PSC/BOClean over on DSLR have come to the - rather logical - conclusion test is crap. For the benefit of readers/posters it's a good thing the message finally arrived ;).

Now, let's wait for a new one to pop up ::) - I sincerely hope this is a "lesson learned" for many.

regards.

paul

Paul Wilders

December 2nd, 2003, 11:42 AM

Jim,

-{ Quote: "Well, I tried to point that out, to no avail. Doesn't matter which AV is one's "favorite"--the test is invalid for all of 'em." }-

Quite so.

-{ Quote: "...but, I was beating my head against the wall..." }-

Not really - the wall cracked in the end, as expected ;)

regards,

paul

Primrose

December 2nd, 2003, 11:47 AM

Well heck guys I will help you out a little here without stepping on anyone's toes. :)

DSLR Security forum is a place where many people do this as stated by WCB ;) In that thread you are discussing.

******************************************

The reason tests like this keep popping up and the reason just about anyone feels free to call themselves AV testers, is because people like you spend 5 days and 7 pages of posts discussing it. So in a sense this does look like National Enquirer. It exists not because it has any validity. It's here because people read it and talk about it around their watercoolers.

http://www.dslreports.com/forum/remark,8663795~mode=flat

*******************************************

Hey, they have to go some place..just ask FF again and all the others who enjoy the affect ;-) and posted instead of just reading.

I did not read VirusP's write up or test data but I know the man well if you want to send him a Christmas Card or reach him by mobile phone.

He told me the beginning of this year he sure would like to get into the testing business as a hobby also and not just collecting. He had reservation in doing this but I encourgaged him. Now before you wacked him too hard..those of you who also know him..know that he is a nice man..collects virus, trojan etc. long before most AV/AT companies got in the business of Security..just like I collect baseball cards when younger and still have enough Mickey Mantle in a safety deposit box if sold would buy a few house(but not now ;) I still like to look at them) and he was always known as a trader. Sometimes one for one depending how rare it was. Fantasic Data Base ( yeh I know you have better ;D )

Many of you might not realize that someone can collect without spreading . But for VirusP and many other who really understand what they have and the genius of some of those badboys as you take them apart. it is a fascination to see how someone else has done it compared to others who have the skill. It is a learning experience. But it does not mean you are going to integrate or manipulate yourself.

I do not think to this day he shares that Data Base with the Current AV/AT developers. But I am not sure.

I am happy to see he is learning also how to do testing. He has done a few so far as we all know..if he sticks with it he will get better. I know he enjoys it. Give him some pointer without the wacks if possible.

Be Well,
John

Gavin - DiamondCS

December 2nd, 2003, 12:17 PM

Ironically, many dont listen to the vendors because they believe they are just trying to sell their product. The reality is that everyone with any sense uses an antivirus product, and most didnt decide to do so by reading a security forum. They were scared because one day their PC freaked out, or because they've been taught about viruses, or their neighbour had a virus, or because their ISP MADE them get rid of something that was spreading from their IP..

But users seem to want to listen to a virus coder or collector, who never cared about the state of a users machine, who probably never freely gives help to others to remove infections just because he CAN - AV and AT people in the know do this all the time, even when busy. Many virus writers speak very lowly of the average user, and think they dont deserve to be online because they are "stupid" enough to open that attachment. Meanwhile AV companies make their software to protect and ensure real work can be done without problems.

Yes thats also someone who collects viruses to trade them with others who collect them, and really ultimately is trying to be the "best" VXer by having the most "samples". And of course this allows the then making "creditable" tests on those "samples" for fame and possibly even money.

Having read many forums and zines I know that there have been plenty of these collectors who are very sneaky. For example, one discussion I read was about how someone was caught out for disassembling a certain virus (dont know the name, wish I could remember everything), and creating tens of variants and sending them into KAV as new viruses. This is of course a zoo sample which has extremely low danger factor. KAV seemingly have no alternative but to add the virus to detection. Then, after KAV detect the new viruses he was adding those to his logs, and trading them for more viruses he "needed". VXers like this are MORE of a problem than the original author of the virus, as modified variants are like edited trojans. The original detection doesn't work, and if any of those who collect this virus decide they want to spread it, most AV's wont detect it straight away, until it starts spreading and everyone gets it. Thats the only real danger, as it could be destructive, but even then if its spreading it will be sent in and shared around (it would become ITW)

So back to those variants.. most have been seen by 2 people. The variant creator, and the AV analyst who added detection. Traders wont always have time to examine the new viruses, and those who have such big collections would have a lot less time trying to manage that collection, sending out viruses and receiving more in return, only to catalogue them again and make new logs, rinse, repeat. Some simply save them in case they want to learn from them later when writing a virus of their own..

Well, sorry if this got a little off track :) I dont believe in the tests myself for more reasons than others because I know how VX tools work. So called FAKES for example are created by including a known scanner's signature for a virus inside an otherwise non viral file. However these are determined by fakescan.dat which is essentially a checksum file of known fakes. It has know real knowledge of whether a file is an actual virus or not and whether or not it runs, and spreads. Here is the first few lines from a fakescan.dat I just grabbed off the net

53f0db48
63d69fb5
00030e63
00049c13
00053f97
00065c8c

In fact CRC32 is so weak, that a REAL virus could have one of these checksums if modified or just by chance, and would be thrown out as a fake.

I guess I just added more unwanted fuel to this fire, but Ive never seen any of these things mentioned before :)

Gavin - DiamondCS

December 2nd, 2003, 12:23 PM

Having read the previous post I understand this person could be a very nice person and have good intentions, but I personally trust an AV vendor more ;D. Nothing against the guy at all, he probably doesnt ever SPREAD them - but I see both sides of the argument and trading is in a sense spreading.

Most AV's dont even trade samples with us, let alone send them out to other VXers sight unseen. VXers who may well infect some helpless users. There is no rules in VXing, 14 year old kids with destruction on their mind can trade for samples of CIH..

Anyway, stay safe :) Most of us here are already quite safe

Primrose

December 2nd, 2003, 04:00 PM

Idid not see any fuel for a fire Wayne ;) all good stuff and soo true. But as I recall in the case of VirusP and conversations we had. ..He was very "put out" with so many tests out there with people using "ZOO " stuff.

:)

Too much zoo and funny stuff out there..too many people playing with them to prove any AV/AT you might be running can not cut the Mustard..and they will prove it to you as a user by sending you a sample..and that is really when the discussion came around about what he would like to see in testing.

Now it is possible thatAV/ AT programmers might work with a zoo to improve heuristics.

But back to VirusP.. ;) There are many kinds of collectors and they do it for many reason..some are selective some are not . But in his case the important ones to use for a test would be those that are or have been in the wild..and infected systems..not this one off thingie...and If a collector is careful WHO he trades with and which ones are out there..that was the orginal basis of his thinking why he would like to try testing.

I can assure you he is qualified to understand what he is doing technically. ;) Gavin picked up on the "nice person" thingie..better I should have said ethical..he has no ties and no need to fudge the numbers for what he would do..but I did not review all he did do for those tests...or how he carried it out.

If the results are way off from what the reliable recent Test center have come up with recently of the vendor products he tested..then I bet no one will ever take him serious.

If he did not notifiy first each vendor he was doing a test on their product before he started much less had it published..then he should look at it all again.

Paul Wilders

December 2nd, 2003, 04:32 PM

John,

Thanks for your additional remarks. All in all, this doesn't change the verdict in regard to this test one bit - no offense intended.

Personally, I for one have no reason at all to doubt the fact VirusP is a nice person - by nature I do believe all people are. Then again: that's not the issue here.

Seems to me all has been said, and there's actually not that much to add.

regards.

paul

Primrose

December 2nd, 2003, 08:32 PM

Totally agree and thanks for you time and space. I test product also..but would never post the result ;-) have a magazine do it for me or try to influence any as to the products they choose.

The infrastructure and the experience needed to do credible testing comes with years of dedication. It is not a one man job :)

To gain respect in doing this you also need the confidence and interface with developers of those products.

Do that..and trust will come.

rodzilla

December 3rd, 2003, 03:45 AM

*
> He told me the beginning of this year he sure would like to get into the testing business as a hobby also and not just collecting. He had reservation in doing this but I encourgaged him.

I hope having his test published doesn't inflate his ego to the point where he thinks he's now up there with the antivirus product testing elite ... because he has a l-o-n-g way to go.

> I do not think to this day he shares that Data Base with the Current AV/AT developers. But I am not sure.

Yeah ... I kinda recall him saying some months ago that he doesn't give his rare and exotic undetectable samples to AV vendors because they don't pay him ... not what I would consider a great attitude for someone who wants to break into the mainstream AV world ... a world in which you wear either a white hat or a black hat ... there are no shades of grey.

> I am happy to see he is learning also how to do testing. He has done a few so far as we all know..if he sticks with it he will get better. I know he enjoys it.

Antony got himself a bad rep in the Usenet AV scene a few years ago ... but that was a few years ago. I believe most people can change for the better, and I believe (with a few restrictions) in giving most people a second chance.

> Give him some pointer without the wacks if possible.

I'd rather see him clean up his testing methodology and produce valid and worthwhile results than see him continue to publish crap which will have him ridiculed by the antivirus industry.

I'll steer Antony along the right path if he's prepared to listen and learn and to put in the hours and effort (a lot of hours and a lot of effort!) required to become a professional and credible AV product tester.
*

rodzilla

December 3rd, 2003, 04:00 AM

*
> and If a collector is careful WHO he trades with and which ones are out there..that was the orginal basis of his thinking why he would like to try testing

I recall Antony saying some months ago, in reply to someone who questioned the integrity of his collection, something along the lines of "The people who gave him the files said they were live viruses, and that's good enough for him".

Sorry ... it's nowhere near good enough!

The fundamental rule in the virus world is "It's either a virus or it's not!" ... and the only way to be 100% certain a file is viral is to execute it in a virgin environment on a susceptible operating system.

> I can assure you he is qualified to understand what he is doing technically.

I don't give a rat's arse what technical qualifications he has ... if his testbed isn't 100% verified and his testing methodology isn't 100% perfect then his results will always be flawed.

> If he did not notifiy first each vendor he was doing a test on their product before he started much less had it published..then he should look at it all again.

Not all that essential. Virus Bulletin invites product submissions ... but drive-by tests are OK provided the virus testbed and testing methodology meet the industry standard.

Antony's test failed on both counts.
*

Primrose

December 3rd, 2003, 09:02 AM

Yup...that is why I tried to chose my words very carefully. ;) being technically qualified does not mean you have the experience. Also I knew many of you wanted to say more and really say it constructively
...with of course your usual summation.

This statement you made...

Not all that essential. Virus Bulletin invites product submissions ... but drive-by tests are OK provided the virus testbed and testing methodology meet the industry standard.

This thread is not really about the sacred cow..but I do not agree with you on that point no matter how you rationalize it.

Also this industry standard thing I see so often come up at these open security forums leaves much to be desired. Some may claim they have been in the business of testing since the Flood and therefore they and a few others have agreed to not disagree on the bed and the method..but where I come from everyone who really cares..signs off on the test standards and documentation.

It needs revisions no matter who claims they have the Holy Grail at this time or 'THE' Test bed.

The year is 2003..time for another conference.

;)

This do it my way or do not do it at all is interesting. But no true professional should really feel threatened.

If you think of your goals as dreams, they will never become real.

rodzilla

December 4th, 2003, 12:17 AM

*
>> Not all that essential. Virus Bulletin invites product submissions ... but drive-by tests are OK provided the virus testbed and testing methodology meet the industry standard.

> This thread is not really about the sacred cow..but I do not agree with you on that point no matter how you rationalize it.

Science, Medicine, Forensics, Engineering, etc ... in fact, just about any profession you can name ... has approved testing methodologies in place. If there isn't a suitable existing methodology available then they create a new one. No professional tester uses "his own" methodology ... at least, not if he wants his findings to be credible.

Academia demands (or should demand) strict adherence to an approved testing methodology if you expect to publish your findings without being ridiculed and ostracized.

By its very definition, an approved testing methodology must be as close to foolproof as possible. Without proper testing methodologies we would see an Apollo 11 disaster (through negligence or oversight) every time the space shuttle was launched, and a new Piltdown Man (through fraud and deception) every week.

Ralph Nader's "Unsafe at any speed" would be laughed out of existence if he published it today. So would Patricia Hoffman's series of "VSum" virus summaries.

Times have changed ... and we must change with them. Viruses are no longer a pimple-on-the-arse-of-your-hard-drive inconvenience ... they're costing the world billions of dollars a year ... and computer users need the best possible information on how to protect themselves..

There is no room for shonky tests!

In antivirus product testing, anything less than perfection is worthless!
*

Primrose

December 4th, 2003, 10:35 AM

Yes that all makes sense ;) I see many use the words perfect..and I counted the times 100% was used in this thread..and other threads.

Those are pretty powerful words and percentages...Equating to the BEST the one an only :D

It leaves no room for "others" in the minds of many who use those terms.. even those who might be at 90% or 80%. ::)

They all become shonky tests or products or developments.

When will people wake up ?

Hugs Rod,

Santa is still wrapping gifts..but the name tags have not been applied.

rodzilla

December 4th, 2003, 11:48 AM

*
> Those are pretty powerful words and percentages...Equating to the BEST the one an only

As I said earlier, by its very definition, an approved testing methodology must be as close to foolproof as possible. "100% perfect" is always the strived for ideal. It's seldom easy to achieve ... but it is achievable.

A daunting number of man-hours would be required to properly validate a sample of each of the many thousands of viruses known today from scratch. Virus Bulletin has a head start ... they've been individually verifying viruses as they appeared since 1989, and now need to verify only new viruses as they appear ... perhaps a few hundred a month. (Even this is a fairly big job.)

The fact that 1/2/3/4 (or even 20) scanners identify a file as viral is not a guarantee that it's a real live virus.

The fact that your mate who knows everything about viruses told you a file is viral is not a guarantee that it's a real live virus.

The only way to guarantee that a file is viral is to verify this yourself by actually infecting something under controlled conditions in a sterile environment ... and that's exactly what Virus Bulletin techs do with each and every virus they use in their tests.

> It leaves no room for "others" in the minds of many who use those terms.. even those who might be at 90% or 80%. They all become shonky tests or products or developments.

If an antivirus product test is shonky in itself then all the results are shonky ... whether a particular product detected 100%, 99.99%, or only 2%.

Virus Bulletin puts its reputation ... a reputation they've worked hard and continuously to maintain for fourteen years ... on the line with every test.

You'll always find some loudmouthed pissant "virus expert" slagging VB testing off in Internet forums or in Usenet ... but in its fourteen year history, not one single solitary reputable antivirus professional has slagged it off. Why do you think that is ?

> When will people wake up ?

Some people never wake up! :)
*

Primrose

December 4th, 2003, 01:31 PM

Well this thread at our forum here all started with Zander the guest ;D ;D with the little pointy thing to DSRL. So here is another one for you. This time you will have a chance to even talk to VirusP.

http://www.dslreports.com/forum/remark,8685090~mode=flat
(http://www.dslreports.com/forum/remark,8685090~mode=flat)

And I am sure we would welcome him here :)

Paul Wilders

December 4th, 2003, 02:43 PM

John,

-{ Quote: "And I am sure we would welcome him here" }-

No, we don't - as for this specific topic is concerned. Two reasons for this:

There's nothing substantial to add to what has been said; the first post in the mentioned thread merely displays annoyance, discredits DSLR mods and various others, without any reason.

regards.

paul

VirusP

December 4th, 2003, 04:06 PM

Ok, so since i ain't welcomed, i just have a couple of questions for you:

1) I have been annoyed by the fact that noone from this forum or another, who has been an av specialist-security specialist even-has emailed me, suggesting ways to improve the quality of the tests i perform. The only thing i have got till now from certain av related ppl is discredibility, disapprovement and bad rep.

2) I don't think everybody knows if and how much related certain forums are to specific av software companies .. let's say i got an av software, or work at such a company for all that i care, and start up a nice little forum, praising "my own" av software. Would that be just?

3) Since VB is the best and most credible av testing org in the world, how come they never publish the vx list they use, or the procedure they follow???

4) Why are certain software ONLY being tested at those tests? I managed to gather-up almost 50 (!!!) antivirus and anti-trojan software, how many of them are included in the VB test? Are the rest of them out of the market? Can't a pc user buy one of them? Why are they excluded afterall?

5) Why do i get the feeling that, like in the av market, people in the av scene do NOT want others to "intrude" and learn the game???

6) If i saw some guy trying to learn a job i am pretty good at, i'd try to help him, unless i felt threatend by the fact that one day he could get my job .. i may be considered a "newbie" compared to many of av experts, nevertheless i do the best i can do. What do they do? Sit in front of their screen and start calling guys like me failures. Now, isn't this all a pretty good reason for me to get upset ???

Best regards to u all

Antony a.k.a. VirusP

Paul Wilders

December 4th, 2003, 05:19 PM

-{ Quote: " quoting: VirusP link=board=24;threadid=17092;start=30#msg106897 date=1070571998]
Ok, so since i ain't welcomed..." }-

Please read carefully: you are welcomed - but not in regard to your test in question.

-{ Quote: "...i just have a couple of questions for you:" }-

right.

-{ Quote: "1) I have been annoyed by the fact that noone from this forum or another, who has been an av specialist-security specialist even-has emailed me, suggesting ways to improve the quality of the tests i perform. The only thing i have got till now from certain av related ppl is discredibility, disapprovement and bad rep." }-

This is about your test, and therefore should not be addressed. As an exception to the rule: As soon as you make your tests public in any way, it's bound to be discussed - all kinds of tests are discussed all over the web. One can't expect people contacting the author - that's not the way it works. You are no exception to this common rule.

As for the reactions: this does come with the territory as well. Logic demands not blaming the one's who did react one way or another. In case there are valid reasons to praise a test, that will be the major consensus. This goes the other way around as well.

-{ Quote: "2) I don't think everybody knows if and how much related certain forums are to specific av software companies .. let's say i got an av software, or work at such a company for all that i care, and start up a nice little forum, praising "my own" av software. Would that be just?" }-

In principal: yes, that would be allowed as long as it's clear for all to see company X is promoting his own software. It's up to the forum visitors to judge. Many of such forums do exist from major software companies, as you are well aware of.

-{ Quote: "3) Since VB is the best and most credible av testing org in the world, how come they never publish the vx list they use, or the procedure they follow???" }-

I'm glad to hear we do have something in common: we both have the same high esteem in regard to VB. As for your question: they are in business since 1988, and cover all ITW viruses, determining them one by one.

-{ Quote: "4) Why are certain software ONLY being tested at those tests? I managed to gather-up almost 50 (!!!) antivirus and anti-trojan software, how many of them are included in the VB test?" }-

The criteria used by VB as well documented on their website, and free for all to examine. As you are well aware of, VB is focussed on antiviruses. Antivirus companies are free in putting their software up for testing. In case they don't show up in the VB tests, they simply have chosen not to submit their software. Antitrojan software is - and never has been - a VB issue.

-{ Quote: "Are the rest of them out of the market? Can't a pc user by one of them? Why are they excluded afterall?" }-

See above.

-{ Quote: "5) Why do i get the feeling that, like in the av market, people in the av scene do NOT want others to "intrude" and learn the game???" }-

I'm in no position to answer that question; it's your feeling. You are the only one who can answer that one.

-{ Quote: "6) If i saw some guy trying to learn a job i am pretty good at, i'd try to help him, unless i felt threatend by the fact that one day he could get my job .. i may be considered a "newbie" compared to many of av experts, nevertheless i do the best i can do. What do they do? Sit in front of their screen and start calling guys like me failures. Now, isn't this all a pretty good reason for me to get upset ???" }-

VirusP, as I see it, no one is questioning your good intentions. But let's make a distinction here: in case you want help in educating yourself: I for one do applaud such an effort, and do wish you all the best in succeeding - ending with a nice job. On the other hand: as long as you are not educated enough, IMO it would be a wise decision not/i] to publish your test(s) - for reasons posted on many forums and boards. This apart from the fact, it's fairly impossible to level with for example VB.

Thus: in case you are upset - and it sure seems like it - don't blame all who have critised you. They have had valid reasons to do so.

That said: I do wish you all the best, educate and end up with a fine [i]white hat job - without trading malware ;)

Finally: I will not allow discussing your test(s) all over again over on this board; there's no use in getting into an endless loop. If necessary, this thread will be closed at the spot.

regards.

paul

VirusP

December 4th, 2003, 05:29 PM

Thank you for your response. It seems like we see things in another perspective, although i agree with you in some matters. Nevertheless, i must respect this forum's rules and i will not make any more comments on this post. If anyone wants, i can be reached through my website (i suppose most of you know it), or even by IM.

Best regards
Antony a.k.a. VirusP

Paul Wilders

December 4th, 2003, 05:43 PM

-{ Quote: "Thank you for your response. It seems like we see things in another perspective, although i agree with you in some matters." }-

You're welcome.

-{ Quote: "Nevertheless, i must respect this forum's rules and i will not make any more comments on this post." }-

Thanks for doing so - I do respect you for that.

-{ Quote: "If anyone wants, i can be reached through my website (i suppose most of you know it), or even by IM." }-

Of course. We might disagree on several issues - but as a registered member over here, you ar welcome to attribute in a constructive way, and use all benefits this board provides.

All the best,

paul

sig

December 4th, 2003, 06:13 PM

In response to VirusP's questions regarding the VB: I'm no expert, but when cruising the VB archives online some time ago I believe I saw that the list of viruses used (both zoo and ITW) in a particular month's tests was indeed available.

From that I deduced that the detailed information re: the viruses used in the tests and the methodology is indeed available, presumably to subscribers to the publication and perhaps only occasionally or rarely in the archives to non subscribers on the website. Perhaps actual subscribers to the VB can confirm or further illuninate this issue.

As to why doesn't the the VB test ffifty programs or at least more than they do test?

1. The VB tests only AV programs.
2. They only test those programs which AV vendors themselves have submitted to them.

Although some posters in the past have claimed that VB chooses what products to test and thus repeatedly denigrated the VB for not testing their product of choice, this is clearly not the case if one actually investigates based on the info available from the mag itself online.

If one actually reads some of the back issues that are available online it's clear that the vendors choose when they will submit a product for testing and, if they have several products, which one(s) they will submit . So clearly it is the AV vendors that determine what products are submitted for testing. The VB will not test a product that has not been submitted by the vendor.

This info is not hidden from public view, one just has to read the mags available online....sometimes some of the archived mags have more info than others since it appears the whole mag isn't always archived online. But after reading several of those available online, this is the info I gleaned. Obviously, paid subscribers get the mag's full contents and the online archive, as far as I can tell, doesn't appear to be completely comprehensive. But still enough information is there to be able to know when some of the VB's loudest critics on the chatboard circuit haven't bothered to check their "facts."