new results from AV-Test.org (Q1/2008)

Discussion in 'other anti-virus software' started by Valentin_Pletzer, Jan 22, 2008.

Thread Status:
Not open for further replies.
  1. Paul Wilders

    Paul Wilders Administrator

    Joined:
    Jul 1, 2001
    Posts:
    12,475
    Location:
    The Netherlands
    I don't have to and I don't have all of them running on various systems. I do have licensed copies, merely for testing on different systems. Comes with the territory ;)

    regards,

    Paul
     
  2. Firecat

    Firecat Registered Member

    Joined:
    Jan 2, 2005
    Posts:
    8,251
    Location:
    The land of no identity :D
    Antony Petrakis has ZERO affiliation with AV-test.org and Andreas Marx. Guaranteed :)

    Secondly, the numbers of malware that are being described by Marx do not seem erroneous to me if we consider the fact that the F-Secure blog article was referring to a different thing (plus 1 signature may cover thousands of infected samples) and that there are many "variants" (repacked versions) of the same malware.

    The test is at least moderately valid IMO. This time they covered more aspects than just the detection rate on-demand which is a good thing. The only real standout for me this time was AVG (which was better than KAV and I was not expecting this); others remained pretty much as they should :)
     
  3. Paul Wilders

    Paul Wilders Administrator

    Joined:
    Jul 1, 2001
    Posts:
    12,475
    Location:
    The Netherlands
    Thanks for the explantion Valentin. That said: guessing is just fine. I for one would like to see the excel sheets coming with all the specs - the common and usual way. There's simply no other way to examine and come to a final conclusion.

    Thanks. Andreas Marx has been a contributing member from this board; I do invite him to sign up as a member once more and provide the necessary info and comments.

    I do applaud you for that.

    Now, here's the essence concerning all this: all of us simply don't have access to essential information from the testing organization. Consequently, in the end we all are in the dark here. Here;s my point: testing organizations should never merely provide results. In case they do want to go public, needed info has to be provided as well so interested ones can make a judgement based on all the facts. On top of that, revealing test results should come from the horses mouth directly. In my humble opinion Marx would have done a far better job going public after the meeting in Spain.

    External sources in effect are - at the least in an ideal world - independent, very skilled organizations specialized in testing software in combo with malware in this context. I do agree that can be an issue indeed. av-test.org/Marx unfortunately does have a history here. Without going into specifics, I do remember at least one controversy between test(s) performed and made public and at least one security software company questioning the way of testing and consequently the result coming from that. And - they we're right. Now, this is not intended as an "av-test.org/Marx bashing". At the most, question marks can and have been all over the web.

    It does matter in a testing environment. It does matter to know wether or not testing has been performed out-of-the-box or software being tweaked to the maximum. It does matter wether samples are or are not detected while executed, instead of being dormant.

    That's a personal choice I'll have to respect. focussed on security; those issues are of far minor importance - at least in my book.

    I do understand. Nothing wrong with a solid discussion ;) . Then again: as stated above, without all the needed info coming from the horse mouth, it does end up in a crippled discussion.

    All the best,

    Paul
     
  4. IBK

    IBK AV Expert

    Joined:
    Dec 22, 2003
    Posts:
    1,886
    Location:
    Innsbruck (Austria)
  5. Paul Wilders

    Paul Wilders Administrator

    Joined:
    Jul 1, 2001
    Posts:
    12,475
    Location:
    The Netherlands
    We may and do have a different opion here, IBK ;)

    That's good to know, but outside the scope from this specific thread.

    My point exactly. Different md5's in essence does not say that much. Hexing a sample in a very minor way results in a different/new md5. Fact remains, it still is one and the sampe sample.

    Again: my point exactly. In my humble view, that's a fine way in determining real new malware, vs. bloating about variants being brand new discovered samples. Fact remains: they are variants, no more, no less. Hyping numbers with this in mind is as far as I'm concerned just that: creating a hype.

    It could well be indeed. ITW-revived and hexed samples can be a threath indeed. Heuristics do come into play here.

    Wrong approach. Marx performed the test - Marx should provide all needed ins and outs needed for all interested ones to come to a fair conclusion - before just drop the results to any third party. In case Chip is to blame, that's merely because they have been eager to go public without having anything to backup the test results.

    In essence, that's a good and needed decision - if only marketing-wise.

    I do hope you have recovered in the meanwhile, and have had a safe and sound trip back home ;)

    regards,

    paul
     
  6. Paul Wilders

    Paul Wilders Administrator

    Joined:
    Jul 1, 2001
    Posts:
    12,475
    Location:
    The Netherlands
    IBK, no offense intended! - but please define "samples". Please provide a link as for where Marx did make that statement. Apart from that: I for one would like to have some real proof from that statement. It's an easy one to state - it's quite a different story to back it up. Then again: Andreas Marx is very welcome (once more) to revive his membership over here and provide info first hand ;)

    regards,

    paul
     
  7. Paul Wilders

    Paul Wilders Administrator

    Joined:
    Jul 1, 2001
    Posts:
    12,475
    Location:
    The Netherlands
  8. Firecat

    Firecat Registered Member

    Joined:
    Jan 2, 2005
    Posts:
    8,251
    Location:
    The land of no identity :D
    *Maybe* this post and the thread in which it was posted (which contains a few more quotes from Marx) might put some insight into this:

    https://www.wilderssecurity.com/showpost.php?p=1011379&postcount=102
     
  9. IBK

    IBK AV Expert

    Joined:
    Dec 22, 2003
    Posts:
    1,886
    Location:
    Innsbruck (Austria)
    @Paul: source -> powerpoint presentation about dynamictesting from marx, page 4 ("Our AV lab is receiving about 2,000 to 2,500 new unique malware samples per hour!"). That paper is not yet on marx website (but he usually puts the papers on his website after some time), but i read this also on some german magazine website at the begin of january (but dunno remember yet the url). i only state what i read and hear, for details or interpretations of what someone means i am not the right person to ask, as i am not related in any way with marx.
     
  10. Kees1958

    Kees1958 Registered Member

    Joined:
    Jul 8, 2006
    Posts:
    5,857
    Yep, but

    In a scientific world your test method is as important as your test result. So scientifically Paul is right. But this is a very aesthetic and old school thinking.

    In daily practice popular TV programs do not even bother to mention their test method anymore. They do not provide observations and evaluation criteria in depth, only their interpretation/personal opinion. The problem with interpretation is, you either believe the source or not (who says so). This is common practise and one of the reason why PCMag test/reviews for instance are bashed by enthousiasts at Wilders, but noob readers will take their opinion as true, because they trust the source.

    Now in the latest years the consumer feedback reports are very popular. When you are looking for a new digital camera, you can find sites which rate them based on results in tests and personal experiences of individual customers. They just reflect the consumer opinion.

    Now Paul, instead of behaving old school with a (scientifically) correct argumentation, face the way the internet changes consumption and information handling: go new age.

    Provide some stickes/polls with Antivirus and HIPS (all sort of security aps) where posters are allowed to enter an opinion. When you look at www.twekers.net people can only post evaluations when they earned good vibes (for instance posts at wilders). Tweakers has found a nice mechanisme to keep negative trolls out (not able to evaluate a supplier).

    Paul you are right, but it ddoes not going tochange the way test organisations are publishing test results of security aps. They need air time, scoops and headline references. They willl become less scientific, So let's innovate Wilders FORUM (with consumer evaluations: for instance you got an infecton using AV of the brannd 'nuts' or brand 'monkey'

    Lusher already propsed a poll, https://www.wilderssecurity.com/showpost.php?p=1164955&postcount=62
     
    Last edited: Jan 23, 2008
  11. Inspector Clouseau

    Inspector Clouseau AV Expert

    Joined:
    Apr 2, 2006
    Posts:
    1,329
    Location:
    Maidenhead, UK
    Correct. Let's continue with your example. Just assume that someone who was using a Canon Powershot A<number here> gets his hands down on a Canon Rebel Xti (400D). I can gurantee you that he will write that this is the best camera on earth. (Because he compared it directly to his previous one) Does his opinion make that the best camera for a professional photographer? For sure not, because if he goes Canon Line then he'll pick a EOS 1 with some proper L-Lenses (The red ones). The difference is here: *THIS* guy knows already exactly what *HE* needs, because he's working in that field. HOWEVER; all the readers with no "clue" will start telling their neighbors there's nothing better than a 400D with standard lenses until they see some "better" results. Got the message? :D
     
  12. Kees1958

    Kees1958 Registered Member

    Joined:
    Jul 8, 2006
    Posts:
    5,857
    Yes, but therefore you should make it like who saved your (censored), what package did not protect you from infection.

    Dear inspector clouseau this is correct. Pshychologist have proven that before you are buying something you might rate A versus B as 49% to 51%. Shortly after the purchase this will shift to 43% to 57% and three months later it will be 30% versus 70%. So even taking this distortion into account, people are seriously looking at those user experience sites.

    So although I just told a arguments against my statement (meaning user experience are unreliable and only related to their own previous experience), daily practise add prrof to my statement: lets innovate wilders security with experience polls (like lusher suggested)
     
  13. Inspector Clouseau

    Inspector Clouseau AV Expert

    Joined:
    Apr 2, 2006
    Posts:
    1,329
    Location:
    Maidenhead, UK
    It goes even more beyond this... Let's ask a very straight forward question. Would you admit to other people that you bought crap? (Besides Family Members...) Ofc you try to avoid that because it makes you look "stupid" 'coz they assume u were not evaluating/researching it before you bought it. Thus, you try to "promote" your PERSONAL selection even with the fact that you know it is NOT perfect as you would like to have it. I'll tell you something...

    *IF* i would ever believe in PUBLIC made (by different, unknown people) tests i'd be driving around with a "LEXUS OWNERS CLUB" sticker on my car, even if it's not a lexus.
     

    Attached Files:

  14. Miyagi

    Miyagi Registered Member

    Joined:
    Mar 12, 2005
    Posts:
    426
    Location:
    None
    BTW - Looks like a Subaru. :p
     
  15. C.S.J

    C.S.J Massive Poster

    Joined:
    Oct 16, 2006
    Posts:
    5,029
    Location:
    this forum is biased!
    good to see you back posting IC,

    so are you in agreement of Paul or not? :blink:

    and whats your general idea about such huge tests?
     
  16. Inspector Clouseau

    Inspector Clouseau AV Expert

    Joined:
    Apr 2, 2006
    Posts:
    1,329
    Location:
    Maidenhead, UK
    Could be, i found that picture in some discussion where they wrote that this sticker attracts womans. As long as the windows are black and no door opens there is for most of the drivers quite a good chance that this actually might work for a few sec.
     
  17. Miyagi

    Miyagi Registered Member

    Joined:
    Mar 12, 2005
    Posts:
    426
    Location:
    None
    How about viagra? :D

    Back on topic, even though my F-PROT was not on the ++ side, it sure is improving. Keep up the good work!
     
  18. Kees1958

    Kees1958 Registered Member

    Joined:
    Jul 8, 2006
    Posts:
    5,857
    YES, I once bought new oversized pitons in my racer. It just did not work out. Because I shared this info, my dealer found a few cheap race exhausts, which might be a solution but were of a different model. I bought them and I am very very very very very very very very happy now,

    NOOP, and it does not work that way in general. There is a satisfaction threshold which turns dissatisfied consumers into real trolls and determined product bashers.

    There is applicable formula which skips out the average bandwith and only takes the very very satisfied users and the disapointed users into account. When a 10 was great and below a 6 is unsatisfactory. The 5 to 8 ratings should be skipped on average.

    But let's switch to a consumer behavioral forum where marketeers and psychologist take part in the discussion, not on Wilders Security.
     
    Last edited: Jan 23, 2008
  19. Inspector Clouseau

    Inspector Clouseau AV Expert

    Joined:
    Apr 2, 2006
    Posts:
    1,329
    Location:
    Maidenhead, UK
    As you prolly noticed paul is still alive and posting that means i didn't have my argument with him *yet*. :D

    But i can confirm (or lets better call it backup) that you can easily reach such number of samples within 6 months. Sure, you will have huge amounts of the SAME or at least VERY SIMILAR virus group (example: Tibs) *BUT* (and that is the important point (!) they are from a point of binary DIFFERENT from each other (poly layer etc). There are hundred thousands of new backdoors (technical note... we shouldn't call them *NEW*, but "Variant") of all "RMPITB" Trojans/Backdoors such as Hupigon's. To give you some idea... Last month alone 1820 DIFFERENT Password Stealers for the Online Game LINAGE. Go figure. One Game - 1820 malwares in ONE month. All chinese binaries.
     
  20. Dark Shadow

    Dark Shadow Registered Member

    Joined:
    Oct 11, 2007
    Posts:
    4,553
    Location:
    USA
    Ok If these test are totally Honest and accurate, How does avast score a higher percentage then nod or drweb with no heuristic In place other then there email scanning? and If these test where all repeated again with all the Exact same settings and procedure would the test reveal the exact same results.Please dont get me wrong I have always liked avast and thought of it as a quality well rounded program and always seems to be getting better.I am Just curious How such strong heuristic scanners as nod and drweb can fall behind Avast pecentage wise.
     
  21. C.S.J

    C.S.J Massive Poster

    Joined:
    Oct 16, 2006
    Posts:
    5,029
    Location:
    this forum is biased!
    i really do question the 99% or whatever AV's, about what they are really adding to their database. :rolleyes:
     
  22. lucas1985

    lucas1985 Retired Moderator

    Joined:
    Nov 9, 2006
    Posts:
    4,047
    Location:
    France, May 1968
    And "RMPITB" means what? Thanks and I'm glad to see you posting again :)
     
  23. Paul Wilders

    Paul Wilders Administrator

    Joined:
    Jul 1, 2001
    Posts:
    12,475
    Location:
    The Netherlands
    Fair enough, Andreas ;)

    regards,

    paul
     
  24. Paul Wilders

    Paul Wilders Administrator

    Joined:
    Jul 1, 2001
    Posts:
    12,475
    Location:
    The Netherlands
    Grin :D

    No argument here.

    In essence: this is true in various cases.

    Variants indeed ;)

    I for one won't argue that. The overall issue concerning testing as well as revealing all needed info coming with it still holds.
     
  25. Paul Wilders

    Paul Wilders Administrator

    Joined:
    Jul 1, 2001
    Posts:
    12,475
    Location:
    The Netherlands
    ...and as far as I'm concerned, we'll stick to the approach mentioned above. Just call me old fashioned ;) . We're not in the "consumer business" and the tactics coming with that. In case people do look for the easy way out: "most votes do count, whatever the reason", they are on the wrong spot over here anyway.

    regards,

    paul
     
Thread Status:
Not open for further replies.
  1. This site uses cookies to help personalise content, tailor your experience and to keep you logged in if you register.
    By continuing to use this site, you are consenting to our use of cookies.