VirusP test in PC Utilities

It's got me stumped why a computer magazine would run a virus spreader's crappy AV tests, but one just did. The usual people are praising it on DSLReports Forums. http://www.dslreports.com/forum/remark,8620641








Added URL tags

 

Couldn't agree more!


tECHNODROME

 

<snipped>

And yes, this "test" is far from what it claims to be. I'd look elsewhere for my information if I were a reader of this magazine.


LWM - snipped out the first paragraph. It really doesn't help to comment on other posters and their opinions... That just leads to more personal attacks...

 

I read your comment to Vampirefo on DSL .........
=====
Vampirefo>> "Here is a challenge for you get your AV Company to offer $5.00 a piece for each virus they missed on VirusP's test. They only have to pay if EICAR says they are indeed viruses, no samples will be given to the AV company until money changes hands."

Zander> I've got a better idea, that will put an end to this argument, and if you're right, it won't cost you a cent. According to you, they are all viruses, so put your money where your mouth is and pay EICAR $5 each for every file in the collection that is *not* a virus.
=====

You shoot a good game of pool, Minnesota Fats!

 

> Not all of them do praise this useless "test"

Most of the "in crowd" seem to think it's great.

> and as far as I know, the author is a virus collector, not a virus spreader

What's the difference ? "Collectors" (who are outside the antivirus industry) will trade viruses with anyone who comes along, with no regard for who they are or what they might do with the viruses.

> and certainly no tester.

Yep.

> Personally, I would recommend everyone to disregard such "tests".

I have little regard for any commercial computer magazine test apart from the few which utilize the resources of Virus Bulletin ... in my opinion they do a disservice to their readers and are a waste of the paper they're printed on ... and to pre-empt the predictable "You like Virus Bulletin because NOD32 is always #1 in its tests" wails from the peanut gallery, I'll point out that I'm on record as backing Virus Bulletin as the leading independent antivirus product tester all the way back to 1989.

 

> And yes, this "test" is far from what it claims to be. I'd look elsewhere for my information if I were a reader of this magazine.

Right!

My opinion is, and has been for years, that only three antivirus product tests (and a few commercial computer magazine comparative reviews which utilize their resources) have any value in the real world. Virus Bulletin is #1 ... ICSA and Checkmark can fight it out for the #2 slot ... the rest have value only to tight little cliques of lemming-like sycophants who would worship Dark Avenger if he posted an antivirus product review.

(Roman (Rokop) is shaping up to be a serious contender ... and if he keeps improving his test bed and methodology then I might even include his tests in my "worthwhile" category sometime down the track.)

Having said that, there are a few (very few!) guys who I've known for years (Paul Wilders, Alex Byron, and Jan Wikstrom, are three who spring to mind) who know viruses and know antivirus software and know how to test it properly. I consider their tests valid and worthy of consideration (as a guide) because they put in the time and effort required to ensure that they test against 100% verified infectious files. Their tests are limited to some degree because their test beds are limited (in numbers) ... but they all test products against real live viruses which are out and about and are most likely to bite you at the time of the test ... not against VX "collections" containing hundreds/thousands of antique DOS viruses, boot sector images, lab samples, collector's one-offs, "simulated viruses", and unknown amounts of non-replicating crud.

[Soapbox mode OFF]

 

>> Most of the "in crowd" seem to think it's great.

> No offense to them, but in my view "most of the crowd" never has been or will be a decive criterium for obvious reasons.

Unfortunately, empty vessels make the loudest noise.

>> What's the difference ? "Collectors" (who are outside the antivirus industry) will trade viruses with anyone who comes along, with no regard for who they are or what they might do with the viruses.

> Although I agree generally spoken this is true, I for one are giving the benefit of the doubt here, as long as there are no facts this person actually doe trade viruses.

Do a Google search for "VirusP". The very first entry is "VirusP - VX Trading page - Virus collector".

"Trading" = "swapping" = "spreading" ... not "spreading" as in "spreading maliciously over the Internet" or whatever, but "spreading" nonetheless.

>> I have little regard for any commercial computer magazine test apart from the few which utilize the resources of Virus Bulletin ... in my opinion they do a disservice to their readers and are a waste of the paper they're printed on ...

> Harshly put

Harsh ... perhaps. Accurate ... definitely!

> but indeed it's for good reasons all serious and major AV companies submit their product(s) themselves for VB testing.

There is a small clique of "experts" on the security forums who ridicule Virus Bulletin at every opportunity, but they seem to forget that the VB100 is the award every antivirus vendor strives to win. (Virus Bulletin uses this as a slogan. They pinched it from me!)

 

>> Harsh ... perhaps. Accurate ... definitely!

> Grin...you've been known for your unique personal way of phrasing

I'm really a very modest guy.

>> There is a small clique of "experts" on the security forums who ridicule Virus Bulletin at every opportunity

> What else is new? I'm pretty sure their influence is hardly of any importance - especially in the big picture, and that's by no means security forums...

I get a laugh out of watching all the little dramas unfold. Every now and then a real IT security professional turns up on DSL, but they seldom stick around for long. Some German IT wizard made some convincing arguments against taking VirusP's previous test seriously some months ago, but he vanished after less than a week. Tetsu-ko would certainly know more about IT security (and viruses) than all the DSL regulars combined ... she's in a class of her own ... but the resident "experts" managed to piss her off in just a couple of days.

>> ...but they seem to forget that the VB100 is the award every antivirus vendor strives to win.

> Quite so, as I stated above.

Yep ... you did.

>> (Virus Bulletin uses this as a slogan. They pinched it from me!)

> Ever considered copyright?

I posted it all over the place, so I guess they figured it was Public Domain.

I don't mind ... it's good for my ego.

 

>> I get a laugh out of watching all the little dramas unfold. Every now and then a real IT security professional turns up on DSL, but they seldom stick around for long. Some German IT wizard made some convincing arguments against taking VirusP's previous test seriously some months ago, but he vanished after less than a week. Tetsu-ko would certainly know more about IT security (and viruses) than all the DSL regulars combined ... she's in a class of her own ... but the resident "experts" managed to piss her off in just a couple of days.

> Well, I'm (and that's a board policy) am not into bashing other boards/forums - and that goes for DSL as well.

Yep ... inter-forum wars are a pointless waste of time and bandwidth.

> In all honesty, one can't blame Justin or WCB for the conduct from their audience/posters.

I guess they try the best they can under trying conditions.

> In my opinion there are quite alot of people over there having lots of expertise on various issues.

Unfortunately the whispers of the people with real expertise are often drowned out by the shouts of the wannabes.

I just took a look at http://www.virus.gr/english/fullxml/default.asp?id=62&mnu=62

Obviously the True Believers haven't read it closely ... or they don't understand it.

VirusP> "The 58306 virus samples were chosen using VS2000 according to Kaspersky, F-Prot, RAV, Nod32, Dr.Web and McAfee antivirus programs. Each virus sample was unique by virus name, meaning that AT LEAST 1 antivirus program detected it as a new virus."

In plain English, what VirusP is saying is that if AT LEAST 1 of either Kaspersky or F-Prot or RAV or Nod32 or Dr.Web or McAfee tagged a virus as "live" then that was good enough verification for him.

Sorry ... it's nowhere near good enough for me ... and it's nowhere near good enough for the antivirus industry to recognize the test as valid.

VirusP> All "fake" virus samples were removed, as well as "garbage" files.

What methodology did VirusP use to determine which samples were "fake" viruses and which files were "garbage" ? Scan them with AT LEAST 1 of either Kaspersky or F-Prot or RAV or Nod32 or Dr.Web or McAfee ? ROFL

Disassembly will show with almost 100% accuracy that a given file is viral, but the only way to verify viral activity beyond all doubt is to execute the file on a susceptible operating system. If it's infectious then it's a virus. If it's not infectious then it's not a virus. No other methodology is acceptable!

VirusP> The virus samples were divided into these categories, according to the type of the virus :
[ . . . ]
VirusP> Malware = DoS, Constructors, Exploit, Flooders, Hoax, Jokes, Nukers, Sniffers, Spoofers, Virus Construction Tools, Virus Tools, Corrupted, Droppers, Intended, PolyEngines.

Wait a minute! Didn't VirusP just say All "fake" virus samples were removed, as well as "garbage" files. ?

If used in an antivirus test, what are Flooders, Hoaxes, Jokes, Nukers, Sniffers, Spoofers, Corrupted viruses, and Intended viruses if not "garbage" ?

They're certainly not the verified live viruses which must be used to the exclusion of everything else if a test of antivirus programs is to have any credibility!

Still, it could have been worse ... VirusP could have included a few hundred .ba$ files!

 

Well, I tried to point that out, to no avail. Doesn't matter which AV is one's "favorite"--the test is invalid for all of 'em.

...but, I was beating my head against the wall...

 

Well heck guys I will help you out a little here without stepping on anyone's toes.

DSLR Security forum is a place where many people do this as stated by WCB In that thread you are discussing.

******************************************

The reason tests like this keep popping up and the reason just about anyone feels free to call themselves AV testers, is because people like you spend 5 days and 7 pages of posts discussing it. So in a sense this does look like National Enquirer. It exists not because it has any validity. It's here because people read it and talk about it around their watercoolers.

http://www.dslreports.com/forum/remark,8663795~mode=flat

*******************************************

Hey, they have to go some place..just ask FF again and all the others who enjoy the affect ;-) and posted instead of just reading.

I did not read VirusP's write up or test data but I know the man well if you want to send him a Christmas Card or reach him by mobile phone.

He told me the beginning of this year he sure would like to get into the testing business as a hobby also and not just collecting. He had reservation in doing this but I encourgaged him. Now before you wacked him too hard..those of you who also know him..know that he is a nice man..collects virus, trojan etc. long before most AV/AT companies got in the business of Security..just like I collect baseball cards when younger and still have enough Mickey Mantle in a safety deposit box if sold would buy a few house(but not now I still like to look at them) and he was always known as a trader. Sometimes one for one depending how rare it was. Fantasic Data Base ( yeh I know you have better )

Many of you might not realize that someone can collect without spreading . But for VirusP and many other who really understand what they have and the genius of some of those badboys as you take them apart. it is a fascination to see how someone else has done it compared to others who have the skill. It is a learning experience. But it does not mean you are going to integrate or manipulate yourself.

I do not think to this day he shares that Data Base with the Current AV/AT developers. But I am not sure.

I am happy to see he is learning also how to do testing. He has done a few so far as we all know..if he sticks with it he will get better. I know he enjoys it. Give him some pointer without the wacks if possible.


Be Well,
John

 

Ironically, many dont listen to the vendors because they believe they are just trying to sell their product. The reality is that everyone with any sense uses an antivirus product, and most didnt decide to do so by reading a security forum. They were scared because one day their PC freaked out, or because they've been taught about viruses, or their neighbour had a virus, or because their ISP MADE them get rid of something that was spreading from their IP..

But users seem to want to listen to a virus coder or collector, who never cared about the state of a users machine, who probably never freely gives help to others to remove infections just because he CAN - AV and AT people in the know do this all the time, even when busy. Many virus writers speak very lowly of the average user, and think they dont deserve to be online because they are "stupid" enough to open that attachment. Meanwhile AV companies make their software to protect and ensure real work can be done without problems.

Yes thats also someone who collects viruses to trade them with others who collect them, and really ultimately is trying to be the "best" VXer by having the most "samples". And of course this allows the then making "creditable" tests on those "samples" for fame and possibly even money.

Having read many forums and zines I know that there have been plenty of these collectors who are very sneaky. For example, one discussion I read was about how someone was caught out for disassembling a certain virus (dont know the name, wish I could remember everything), and creating tens of variants and sending them into KAV as new viruses. This is of course a zoo sample which has extremely low danger factor. KAV seemingly have no alternative but to add the virus to detection. Then, after KAV detect the new viruses he was adding those to his logs, and trading them for more viruses he "needed". VXers like this are MORE of a problem than the original author of the virus, as modified variants are like edited trojans. The original detection doesn't work, and if any of those who collect this virus decide they want to spread it, most AV's wont detect it straight away, until it starts spreading and everyone gets it. Thats the only real danger, as it could be destructive, but even then if its spreading it will be sent in and shared around (it would become ITW)

So back to those variants.. most have been seen by 2 people. The variant creator, and the AV analyst who added detection. Traders wont always have time to examine the new viruses, and those who have such big collections would have a lot less time trying to manage that collection, sending out viruses and receiving more in return, only to catalogue them again and make new logs, rinse, repeat. Some simply save them in case they want to learn from them later when writing a virus of their own..

Well, sorry if this got a little off track I dont believe in the tests myself for more reasons than others because I know how VX tools work. So called FAKES for example are created by including a known scanner's signature for a virus inside an otherwise non viral file. However these are determined by fakescan.dat which is essentially a checksum file of known fakes. It has know real knowledge of whether a file is an actual virus or not and whether or not it runs, and spreads. Here is the first few lines from a fakescan.dat I just grabbed off the net

53f0db48
63d69fb5
00030e63
00049c13
00053f97
00065c8c

In fact CRC32 is so weak, that a REAL virus could have one of these checksums if modified or just by chance, and would be thrown out as a fake.

I guess I just added more unwanted fuel to this fire, but Ive never seen any of these things mentioned before

 

Having read the previous post I understand this person could be a very nice person and have good intentions, but I personally trust an AV vendor more . Nothing against the guy at all, he probably doesnt ever SPREAD them - but I see both sides of the argument and trading is in a sense spreading.

Most AV's dont even trade samples with us, let alone send them out to other VXers sight unseen. VXers who may well infect some helpless users. There is no rules in VXing, 14 year old kids with destruction on their mind can trade for samples of CIH..

Anyway, stay safe Most of us here are already quite safe

 

Idid not see any fuel for a fire Wayne all good stuff and soo true. But as I recall in the case of VirusP and conversations we had. ..He was very "put out" with so many tests out there with people using "ZOO " stuff.



Too much zoo and funny stuff out there..too many people playing with them to prove any AV/AT you might be running can not cut the Mustard..and they will prove it to you as a user by sending you a sample..and that is really when the discussion came around about what he would like to see in testing.

Now it is possible thatAV/ AT programmers might work with a zoo to improve heuristics.

But back to VirusP.. There are many kinds of collectors and they do it for many reason..some are selective some are not . But in his case the important ones to use for a test would be those that are or have been in the wild..and infected systems..not this one off thingie...and If a collector is careful WHO he trades with and which ones are out there..that was the orginal basis of his thinking why he would like to try testing.

I can assure you he is qualified to understand what he is doing technically. Gavin picked up on the "nice person" thingie..better I should have said ethical..he has no ties and no need to fudge the numbers for what he would do..but I did not review all he did do for those tests...or how he carried it out.

If the results are way off from what the reliable recent Test center have come up with recently of the vendor products he tested..then I bet no one will ever take him serious.

If he did not notifiy first each vendor he was doing a test on their product before he started much less had it published..then he should look at it all again.

 

Totally agree and thanks for you time and space. I test product also..but would never post the result ;-) have a magazine do it for me or try to influence any as to the products they choose.

The infrastructure and the experience needed to do credible testing comes with years of dedication. It is not a one man job

To gain respect in doing this you also need the confidence and interface with developers of those products.


Do that..and trust will come.

 

 
> He told me the beginning of this year he sure would like to get into the testing business as a hobby also and not just collecting. He had reservation in doing this but I encourgaged him.

I hope having his test published doesn't inflate his ego to the point where he thinks he's now up there with the antivirus product testing elite ... because he has a l-o-n-g way to go.

> I do not think to this day he shares that Data Base with the Current AV/AT developers. But I am not sure.

Yeah ... I kinda recall him saying some months ago that he doesn't give his rare and exotic undetectable samples to AV vendors because they don't pay him ... not what I would consider a great attitude for someone who wants to break into the mainstream AV world ... a world in which you wear either a white hat or a black hat ... there are no shades of grey.

> I am happy to see he is learning also how to do testing. He has done a few so far as we all know..if he sticks with it he will get better. I know he enjoys it.

Antony got himself a bad rep in the Usenet AV scene a few years ago ... but that was a few years ago. I believe most people can change for the better, and I believe (with a few restrictions) in giving most people a second chance.

> Give him some pointer without the wacks if possible.

I'd rather see him clean up his testing methodology and produce valid and worthwhile results than see him continue to publish crap which will have him ridiculed by the antivirus industry.

I'll steer Antony along the right path if he's prepared to listen and learn and to put in the hours and effort (a lot of hours and a lot of effort!) required to become a professional and credible AV product tester.
 

 

 
> and If a collector is careful WHO he trades with and which ones are out there..that was the orginal basis of his thinking why he would like to try testing

I recall Antony saying some months ago, in reply to someone who questioned the integrity of his collection, something along the lines of "The people who gave him the files said they were live viruses, and that's good enough for him".

Sorry ... it's nowhere near good enough!

The fundamental rule in the virus world is "It's either a virus or it's not!" ... and the only way to be 100% certain a file is viral is to execute it in a virgin environment on a susceptible operating system.

> I can assure you he is qualified to understand what he is doing technically.

I don't give a rat's arse what technical qualifications he has ... if his testbed isn't 100% verified and his testing methodology isn't 100% perfect then his results will always be flawed.

> If he did not notifiy first each vendor he was doing a test on their product before he started much less had it published..then he should look at it all again.

Not all that essential. Virus Bulletin invites product submissions ... but drive-by tests are OK provided the virus testbed and testing methodology meet the industry standard.

Antony's test failed on both counts.