VirusP test in PC Utilities

Discussion in 'other anti-virus software' started by Zander, Dec 1, 2003.

Thread Status:
Not open for further replies.
  1. Zander

    Zander Guest

  2. Paul Wilders

    Paul Wilders Administrator

    Joined:
    Jul 1, 2001
    Posts:
    12,472
    Location:
    The Netherlands
    Zander,

    Not all of them do praise this useless "test" - and as far as I know, the author is a virus collector, not a virus spreader - and certainly no tester ;). Personally, I would recommend everyone to disregard such "tests".

    regards.

    paul
     
  3. Technodrome

    Technodrome Security Expert

    Joined:
    Feb 13, 2002
    Posts:
    2,140
    Location:
    New York
    Couldn't agree more!


    tECHNODROME
     
  4. dos

    dos Registered Member

    Joined:
    Oct 17, 2003
    Posts:
    43
    <snipped>

    And yes, this "test" is far from what it claims to be. I'd look elsewhere for my information if I were a reader of this magazine.


    LWM - snipped out the first paragraph. It really doesn't help to comment on other posters and their opinions... That just leads to more personal attacks...
     
  5. rodzilla

    rodzilla Registered Member

    Joined:
    Jun 15, 2002
    Posts:
    653
    Location:
    australia
    I read your comment to Vampirefo on DSL .........
    =====
    Vampirefo>> "Here is a challenge for you get your AV Company to offer $5.00 a piece for each virus they missed on VirusP's test. They only have to pay if EICAR says they are indeed viruses, no samples will be given to the AV company until money changes hands."

    Zander> I've got a better idea, that will put an end to this argument, and if you're right, it won't cost you a cent. According to you, they are all viruses, so put your money where your mouth is and pay EICAR $5 each for every file in the collection that is *not* a virus.
    =====

    You shoot a good game of pool, Minnesota Fats! :D
     
  6. rodzilla

    rodzilla Registered Member

    Joined:
    Jun 15, 2002
    Posts:
    653
    Location:
    australia
    > Not all of them do praise this useless "test"

    Most of the "in crowd" seem to think it's great.

    > and as far as I know, the author is a virus collector, not a virus spreader

    What's the difference ? "Collectors" (who are outside the antivirus industry) will trade viruses with anyone who comes along, with no regard for who they are or what they might do with the viruses.

    > and certainly no tester.

    Yep.

    > Personally, I would recommend everyone to disregard such "tests".

    I have little regard for any commercial computer magazine test apart from the few which utilize the resources of Virus Bulletin ... in my opinion they do a disservice to their readers and are a waste of the paper they're printed on ... and to pre-empt the predictable "You like Virus Bulletin because NOD32 is always #1 in its tests" wails from the peanut gallery, I'll point out that I'm on record as backing Virus Bulletin as the leading independent antivirus product tester all the way back to 1989.
     
  7. Paul Wilders

    Paul Wilders Administrator

    Joined:
    Jul 1, 2001
    Posts:
    12,472
    Location:
    The Netherlands
    Rod,

    Most of the "in crowd" seem to think it's great.

    No offense to them, but in my view "most of the crowd" never has been or will be a decive criterium for obvious reasons.


    What's the difference ? "Collectors" (who are outside the antivirus industry) will trade viruses with anyone who comes along, with no regard for who they are or what they might do with the viruses.


    Although I agree generally spoken this is true, I for one are giving the benefit of the doubt here, as long as there are no facts this person actually doe trade viruses.


    I have little regard for any commercial computer magazine test apart from the few which utilize the resources of Virus Bulletin ... in my opinion they do a disservice to their readers and are a waste of the paper they're printed on ...


    Harshly put - but indeed it's for good reasons all serious and major AV companies submit their product(s) themselves for VB testing.

    regards,

    paul
     
  8. rodzilla

    rodzilla Registered Member

    Joined:
    Jun 15, 2002
    Posts:
    653
    Location:
    australia
    > And yes, this "test" is far from what it claims to be. I'd look elsewhere for my information if I were a reader of this magazine.

    Right!

    My opinion is, and has been for years, that only three antivirus product tests (and a few commercial computer magazine comparative reviews which utilize their resources) have any value in the real world. Virus Bulletin is #1 ... ICSA and Checkmark can fight it out for the #2 slot ... the rest have value only to tight little cliques of lemming-like sycophants who would worship Dark Avenger if he posted an antivirus product review.

    (Roman (Rokop) is shaping up to be a serious contender ... and if he keeps improving his test bed and methodology then I might even include his tests in my "worthwhile" category sometime down the track.)

    Having said that, there are a few (very few!) guys who I've known for years (Paul Wilders, Alex Byron, and Jan Wikstrom, are three who spring to mind) who know viruses and know antivirus software and know how to test it properly. I consider their tests valid and worthy of consideration (as a guide) because they put in the time and effort required to ensure that they test against 100% verified infectious files. Their tests are limited to some degree because their test beds are limited (in numbers) ... but they all test products against real live viruses which are out and about and are most likely to bite you at the time of the test ... not against VX "collections" containing hundreds/thousands of antique DOS viruses, boot sector images, lab samples, collector's one-offs, "simulated viruses", and unknown amounts of non-replicating crud.

    [Soapbox mode OFF]
     
  9. rodzilla

    rodzilla Registered Member

    Joined:
    Jun 15, 2002
    Posts:
    653
    Location:
    australia
    >> Most of the "in crowd" seem to think it's great.

    > No offense to them, but in my view "most of the crowd" never has been or will be a decive criterium for obvious reasons.

    Unfortunately, empty vessels make the loudest noise. :)

    >> What's the difference ? "Collectors" (who are outside the antivirus industry) will trade viruses with anyone who comes along, with no regard for who they are or what they might do with the viruses.

    > Although I agree generally spoken this is true, I for one are giving the benefit of the doubt here, as long as there are no facts this person actually doe trade viruses.

    Do a Google search for "VirusP". The very first entry is "VirusP - VX Trading page - Virus collector".

    "Trading" = "swapping" = "spreading" ... not "spreading" as in "spreading maliciously over the Internet" or whatever, but "spreading" nonetheless.

    >> I have little regard for any commercial computer magazine test apart from the few which utilize the resources of Virus Bulletin ... in my opinion they do a disservice to their readers and are a waste of the paper they're printed on ...

    > Harshly put

    Harsh ... perhaps. Accurate ... definitely! :D

    > but indeed it's for good reasons all serious and major AV companies submit their product(s) themselves for VB testing.

    There is a small clique of "experts" on the security forums who ridicule Virus Bulletin at every opportunity, but they seem to forget that the VB100 is the award every antivirus vendor strives to win. (Virus Bulletin uses this as a slogan. They pinched it from me!) :)
     
  10. Godzilla

    Godzilla AV Expert

    Joined:
    Nov 1, 2003
    Posts:
    63
    :D :D :D
     
  11. Paul Wilders

    Paul Wilders Administrator

    Joined:
    Jul 1, 2001
    Posts:
    12,472
    Location:
    The Netherlands
    Rod,

    Unfortunately, empty vessels make the loudest noise.

    You should take the size of the vessel into account as well :D

    Do a Google search for "VirusP". The very first entry is "VirusP - VX Trading page - Virus collector".

    "Trading" = "swapping" = "spreading" ... not "spreading" as in "spreading maliciously over the Internet" or whatever, but "spreading" nonetheless.


    I stand corrected.


    Harsh ... perhaps. Accurate ... definitely!


    Grin...you've been known for your unique personal way of phrasing ;)


    There is a small clique of "experts" on the security forums who ridicule Virus Bulletin at every opportunity


    What else is new? I'm pretty sure their influence is hardly of any importance - especially in the big picture, and that's by no means security forums...

    ...but they seem to forget that the VB100 is the award every antivirus vendor strives to win.

    Quite so, as I stated above.

    (Virus Bulletin uses this as a slogan. They pinched it from me!)

    Ever considered copyright? ;)

    regards.

    paul
     
  12. rodzilla

    rodzilla Registered Member

    Joined:
    Jun 15, 2002
    Posts:
    653
    Location:
    australia
    >> Harsh ... perhaps. Accurate ... definitely!

    > Grin...you've been known for your unique personal way of phrasing

    I'm really a very modest guy. :) :) :)

    >> There is a small clique of "experts" on the security forums who ridicule Virus Bulletin at every opportunity

    > What else is new? I'm pretty sure their influence is hardly of any importance - especially in the big picture, and that's by no means security forums...

    I get a laugh out of watching all the little dramas unfold. Every now and then a real IT security professional turns up on DSL, but they seldom stick around for long. Some German IT wizard made some convincing arguments against taking VirusP's previous test seriously some months ago, but he vanished after less than a week. Tetsu-ko would certainly know more about IT security (and viruses) than all the DSL regulars combined ... she's in a class of her own ... but the resident "experts" managed to piss her off in just a couple of days.

    >> ...but they seem to forget that the VB100 is the award every antivirus vendor strives to win.

    > Quite so, as I stated above.

    Yep ... you did.

    >> (Virus Bulletin uses this as a slogan. They pinched it from me!)

    > Ever considered copyright?

    I posted it all over the place, so I guess they figured it was Public Domain.

    I don't mind ... it's good for my ego. :)
     
  13. Paul Wilders

    Paul Wilders Administrator

    Joined:
    Jul 1, 2001
    Posts:
    12,472
    Location:
    The Netherlands
    I'm really a very modest guy.

    Sure you are :D


    I get a laugh out of watching all the little dramas unfold. Every now and then a real IT security professional turns up on DSL, but they seldom stick around for long. Some German IT wizard made some convincing arguments against taking VirusP's previous test seriously some months ago, but he vanished after less than a week. Tetsu-ko would certainly know more about IT security (and viruses) than all the DSL regulars combined ... she's in a class of her own ... but the resident "experts" managed to piss her off in just a couple of days.


    Well, I'm (and that's a board policy) am not into bashing other boards/forums - and that goes for DSL as well. In all honesty, one can't blame Justin or WCB for the conduct from their audience/posters. In my opinion there are quite alot of people over there having lots of expertise on various issues. Justin and WCB have created a rather nice platform. It wouldn't be fair to judge them for the way some wish to post over there.
     
  14. rodzilla

    rodzilla Registered Member

    Joined:
    Jun 15, 2002
    Posts:
    653
    Location:
    australia
    >> I get a laugh out of watching all the little dramas unfold. Every now and then a real IT security professional turns up on DSL, but they seldom stick around for long. Some German IT wizard made some convincing arguments against taking VirusP's previous test seriously some months ago, but he vanished after less than a week. Tetsu-ko would certainly know more about IT security (and viruses) than all the DSL regulars combined ... she's in a class of her own ... but the resident "experts" managed to piss her off in just a couple of days.

    > Well, I'm (and that's a board policy) am not into bashing other boards/forums - and that goes for DSL as well.

    Yep ... inter-forum wars are a pointless waste of time and bandwidth.

    > In all honesty, one can't blame Justin or WCB for the conduct from their audience/posters.

    I guess they try the best they can under trying conditions.

    > In my opinion there are quite alot of people over there having lots of expertise on various issues.

    Unfortunately the whispers of the people with real expertise are often drowned out by the shouts of the wannabes.

    I just took a look at http://www.virus.gr/english/fullxml/default.asp?id=62&mnu=62

    Obviously the True Believers haven't read it closely ... or they don't understand it.

    VirusP> "The 58306 virus samples were chosen using VS2000 according to Kaspersky, F-Prot, RAV, Nod32, Dr.Web and McAfee antivirus programs. Each virus sample was unique by virus name, meaning that AT LEAST 1 antivirus program detected it as a new virus."

    In plain English, what VirusP is saying is that if AT LEAST 1 of either Kaspersky or F-Prot or RAV or Nod32 or Dr.Web or McAfee tagged a virus as "live" then that was good enough verification for him.

    Sorry ... it's nowhere near good enough for me ... and it's nowhere near good enough for the antivirus industry to recognize the test as valid.

    VirusP> All "fake" virus samples were removed, as well as "garbage" files.

    What methodology did VirusP use to determine which samples were "fake" viruses and which files were "garbage" ? Scan them with AT LEAST 1 of either Kaspersky or F-Prot or RAV or Nod32 or Dr.Web or McAfee ? ROFL

    Disassembly will show with almost 100% accuracy that a given file is viral, but the only way to verify viral activity beyond all doubt is to execute the file on a susceptible operating system. If it's infectious then it's a virus. If it's not infectious then it's not a virus. No other methodology is acceptable!

    VirusP> The virus samples were divided into these categories, according to the type of the virus :
    [ . . . ]
    VirusP> Malware = DoS, Constructors, Exploit, Flooders, Hoax, Jokes, Nukers, Sniffers, Spoofers, Virus Construction Tools, Virus Tools, Corrupted, Droppers, Intended, PolyEngines.

    Wait a minute! Didn't VirusP just say All "fake" virus samples were removed, as well as "garbage" files. ?

    If used in an antivirus test, what are Flooders, Hoaxes, Jokes, Nukers, Sniffers, Spoofers, Corrupted viruses, and Intended viruses if not "garbage" ?

    They're certainly not the verified live viruses which must be used to the exclusion of everything else if a test of antivirus programs is to have any credibility!

    Still, it could have been worse ... VirusP could have included a few hundred .ba$ files! :D
     
  15. JimIT

    JimIT Registered Member

    Joined:
    Jan 22, 2003
    Posts:
    1,035
    Location:
    Denton, Texas
    Well, I tried to point that out, to no avail. Doesn't matter which AV is one's "favorite"--the test is invalid for all of 'em.

    ...but, I was beating my head against the wall... :doubt: ;) :D
     
  16. Paul Wilders

    Paul Wilders Administrator

    Joined:
    Jul 1, 2001
    Posts:
    12,472
    Location:
    The Netherlands
    Rod,

    In short: right on spot ;).

    On a side note: seems like WCB (forum boss DSLR) as well as Kevin M. from PSC/BOClean over on DSLR have come to the - rather logical - conclusion test is crap. For the benefit of readers/posters it's a good thing the message finally arrived ;).

    Now, let's wait for a new one to pop up :rolleyes: - I sincerely hope this is a "lesson learned" for many.

    regards.

    paul
     
  17. Paul Wilders

    Paul Wilders Administrator

    Joined:
    Jul 1, 2001
    Posts:
    12,472
    Location:
    The Netherlands
    Jim,

    Quite so.

    Not really - the wall cracked in the end, as expected ;)

    regards,

    paul
     
  18. Primrose

    Primrose Registered Member

    Joined:
    Sep 21, 2002
    Posts:
    2,743
    Well heck guys I will help you out a little here without stepping on anyone's toes. :)

    DSLR Security forum is a place where many people do this as stated by WCB ;) In that thread you are discussing.

    ******************************************

    The reason tests like this keep popping up and the reason just about anyone feels free to call themselves AV testers, is because people like you spend 5 days and 7 pages of posts discussing it. So in a sense this does look like National Enquirer. It exists not because it has any validity. It's here because people read it and talk about it around their watercoolers.

    http://www.dslreports.com/forum/remark,8663795~mode=flat

    *******************************************

    Hey, they have to go some place..just ask FF again and all the others who enjoy the affect ;-) and posted instead of just reading.

    I did not read VirusP's write up or test data but I know the man well if you want to send him a Christmas Card or reach him by mobile phone.

    He told me the beginning of this year he sure would like to get into the testing business as a hobby also and not just collecting. He had reservation in doing this but I encourgaged him. Now before you wacked him too hard..those of you who also know him..know that he is a nice man..collects virus, trojan etc. long before most AV/AT companies got in the business of Security..just like I collect baseball cards when younger and still have enough Mickey Mantle in a safety deposit box if sold would buy a few house(but not now ;) I still like to look at them) and he was always known as a trader. Sometimes one for one depending how rare it was. Fantasic Data Base ( yeh I know you have better :D )

    Many of you might not realize that someone can collect without spreading . But for VirusP and many other who really understand what they have and the genius of some of those badboys as you take them apart. it is a fascination to see how someone else has done it compared to others who have the skill. It is a learning experience. But it does not mean you are going to integrate or manipulate yourself.

    I do not think to this day he shares that Data Base with the Current AV/AT developers. But I am not sure.

    I am happy to see he is learning also how to do testing. He has done a few so far as we all know..if he sticks with it he will get better. I know he enjoys it. Give him some pointer without the wacks if possible.


    Be Well,
    John
     
  19. Gavin - DiamondCS

    Gavin - DiamondCS Former DCS Moderator

    Joined:
    Feb 10, 2002
    Posts:
    2,080
    Location:
    Perth, Western Australia
    Ironically, many dont listen to the vendors because they believe they are just trying to sell their product. The reality is that everyone with any sense uses an antivirus product, and most didnt decide to do so by reading a security forum. They were scared because one day their PC freaked out, or because they've been taught about viruses, or their neighbour had a virus, or because their ISP MADE them get rid of something that was spreading from their IP..

    But users seem to want to listen to a virus coder or collector, who never cared about the state of a users machine, who probably never freely gives help to others to remove infections just because he CAN - AV and AT people in the know do this all the time, even when busy. Many virus writers speak very lowly of the average user, and think they dont deserve to be online because they are "stupid" enough to open that attachment. Meanwhile AV companies make their software to protect and ensure real work can be done without problems.

    Yes thats also someone who collects viruses to trade them with others who collect them, and really ultimately is trying to be the "best" VXer by having the most "samples". And of course this allows the then making "creditable" tests on those "samples" for fame and possibly even money.

    Having read many forums and zines I know that there have been plenty of these collectors who are very sneaky. For example, one discussion I read was about how someone was caught out for disassembling a certain virus (dont know the name, wish I could remember everything), and creating tens of variants and sending them into KAV as new viruses. This is of course a zoo sample which has extremely low danger factor. KAV seemingly have no alternative but to add the virus to detection. Then, after KAV detect the new viruses he was adding those to his logs, and trading them for more viruses he "needed". VXers like this are MORE of a problem than the original author of the virus, as modified variants are like edited trojans. The original detection doesn't work, and if any of those who collect this virus decide they want to spread it, most AV's wont detect it straight away, until it starts spreading and everyone gets it. Thats the only real danger, as it could be destructive, but even then if its spreading it will be sent in and shared around (it would become ITW)

    So back to those variants.. most have been seen by 2 people. The variant creator, and the AV analyst who added detection. Traders wont always have time to examine the new viruses, and those who have such big collections would have a lot less time trying to manage that collection, sending out viruses and receiving more in return, only to catalogue them again and make new logs, rinse, repeat. Some simply save them in case they want to learn from them later when writing a virus of their own..

    Well, sorry if this got a little off track :) I dont believe in the tests myself for more reasons than others because I know how VX tools work. So called FAKES for example are created by including a known scanner's signature for a virus inside an otherwise non viral file. However these are determined by fakescan.dat which is essentially a checksum file of known fakes. It has know real knowledge of whether a file is an actual virus or not and whether or not it runs, and spreads. Here is the first few lines from a fakescan.dat I just grabbed off the net

    53f0db48
    63d69fb5
    00030e63
    00049c13
    00053f97
    00065c8c

    In fact CRC32 is so weak, that a REAL virus could have one of these checksums if modified or just by chance, and would be thrown out as a fake.

    I guess I just added more unwanted fuel to this fire, but Ive never seen any of these things mentioned before :)
     
  20. Gavin - DiamondCS

    Gavin - DiamondCS Former DCS Moderator

    Joined:
    Feb 10, 2002
    Posts:
    2,080
    Location:
    Perth, Western Australia
    Having read the previous post I understand this person could be a very nice person and have good intentions, but I personally trust an AV vendor more :D. Nothing against the guy at all, he probably doesnt ever SPREAD them - but I see both sides of the argument and trading is in a sense spreading.

    Most AV's dont even trade samples with us, let alone send them out to other VXers sight unseen. VXers who may well infect some helpless users. There is no rules in VXing, 14 year old kids with destruction on their mind can trade for samples of CIH..

    Anyway, stay safe :) Most of us here are already quite safe
     
  21. Primrose

    Primrose Registered Member

    Joined:
    Sep 21, 2002
    Posts:
    2,743
    Idid not see any fuel for a fire Wayne ;) all good stuff and soo true. But as I recall in the case of VirusP and conversations we had. ..He was very "put out" with so many tests out there with people using "ZOO " stuff.

    :)

    Too much zoo and funny stuff out there..too many people playing with them to prove any AV/AT you might be running can not cut the Mustard..and they will prove it to you as a user by sending you a sample..and that is really when the discussion came around about what he would like to see in testing.

    Now it is possible thatAV/ AT programmers might work with a zoo to improve heuristics.

    But back to VirusP.. ;) There are many kinds of collectors and they do it for many reason..some are selective some are not . But in his case the important ones to use for a test would be those that are or have been in the wild..and infected systems..not this one off thingie...and If a collector is careful WHO he trades with and which ones are out there..that was the orginal basis of his thinking why he would like to try testing.

    I can assure you he is qualified to understand what he is doing technically. ;) Gavin picked up on the "nice person" thingie..better I should have said ethical..he has no ties and no need to fudge the numbers for what he would do..but I did not review all he did do for those tests...or how he carried it out.

    If the results are way off from what the reliable recent Test center have come up with recently of the vendor products he tested..then I bet no one will ever take him serious.

    If he did not notifiy first each vendor he was doing a test on their product before he started much less had it published..then he should look at it all again.
     
  22. Paul Wilders

    Paul Wilders Administrator

    Joined:
    Jul 1, 2001
    Posts:
    12,472
    Location:
    The Netherlands
    John,

    Thanks for your additional remarks. All in all, this doesn't change the verdict in regard to this test one bit - no offense intended.

    Personally, I for one have no reason at all to doubt the fact VirusP is a nice person - by nature I do believe all people are. Then again: that's not the issue here.

    Seems to me all has been said, and there's actually not that much to add.

    regards.

    paul
     
  23. Primrose

    Primrose Registered Member

    Joined:
    Sep 21, 2002
    Posts:
    2,743
    Totally agree and thanks for you time and space. I test product also..but would never post the result ;-) have a magazine do it for me or try to influence any as to the products they choose.

    The infrastructure and the experience needed to do credible testing comes with years of dedication. It is not a one man job :)

    To gain respect in doing this you also need the confidence and interface with developers of those products.


    Do that..and trust will come.
     
  24. rodzilla

    rodzilla Registered Member

    Joined:
    Jun 15, 2002
    Posts:
    653
    Location:
    australia
     
    > He told me the beginning of this year he sure would like to get into the testing business as a hobby also and not just collecting. He had reservation in doing this but I encourgaged him.

    I hope having his test published doesn't inflate his ego to the point where he thinks he's now up there with the antivirus product testing elite ... because he has a l-o-n-g way to go.

    > I do not think to this day he shares that Data Base with the Current AV/AT developers. But I am not sure.

    Yeah ... I kinda recall him saying some months ago that he doesn't give his rare and exotic undetectable samples to AV vendors because they don't pay him ... not what I would consider a great attitude for someone who wants to break into the mainstream AV world ... a world in which you wear either a white hat or a black hat ... there are no shades of grey.

    > I am happy to see he is learning also how to do testing. He has done a few so far as we all know..if he sticks with it he will get better. I know he enjoys it.

    Antony got himself a bad rep in the Usenet AV scene a few years ago ... but that was a few years ago. I believe most people can change for the better, and I believe (with a few restrictions) in giving most people a second chance.

    > Give him some pointer without the wacks if possible.

    I'd rather see him clean up his testing methodology and produce valid and worthwhile results than see him continue to publish crap which will have him ridiculed by the antivirus industry.

    I'll steer Antony along the right path if he's prepared to listen and learn and to put in the hours and effort (a lot of hours and a lot of effort!) required to become a professional and credible AV product tester.
     
     
  25. rodzilla

    rodzilla Registered Member

    Joined:
    Jun 15, 2002
    Posts:
    653
    Location:
    australia
     
    > and If a collector is careful WHO he trades with and which ones are out there..that was the orginal basis of his thinking why he would like to try testing

    I recall Antony saying some months ago, in reply to someone who questioned the integrity of his collection, something along the lines of "The people who gave him the files said they were live viruses, and that's good enough for him".

    Sorry ... it's nowhere near good enough!

    The fundamental rule in the virus world is "It's either a virus or it's not!" ... and the only way to be 100% certain a file is viral is to execute it in a virgin environment on a susceptible operating system.

    > I can assure you he is qualified to understand what he is doing technically.

    I don't give a rat's arse what technical qualifications he has ... if his testbed isn't 100% verified and his testing methodology isn't 100% perfect then his results will always be flawed.

    > If he did not notifiy first each vendor he was doing a test on their product before he started much less had it published..then he should look at it all again.

    Not all that essential. Virus Bulletin invites product submissions ... but drive-by tests are OK provided the virus testbed and testing methodology meet the industry standard.

    Antony's test failed on both counts.
     
     
Loading...
Thread Status:
Not open for further replies.