Interesting ShawdowServer results

Discussion in 'other anti-virus software' started by Bunkhouse Buck, Jun 14, 2008.

Thread Status:
Not open for further replies.
  1. Bunkhouse Buck

    Bunkhouse Buck Registered Member

    Joined:
    May 29, 2007
    Posts:
    1,056
    Location:
    Las Vegas
  2. C.S.J

    C.S.J Massive Poster

    Joined:
    Oct 16, 2006
    Posts:
    5,029
    Location:
    this forum is biased!
    id love to know what they have changed.

    for the past 12+ months, drweb scored 90+ every single day.

    yet, for the last 3-4 weeks, drweb scores 2-8%.

    ive noticed these changes, since the addition of other AV's too.

    id love to hear some 'official' comments from them about it.
     
  3. Coolio10

    Coolio10 Registered Member

    Joined:
    Sep 1, 2006
    Posts:
    1,124
    Wow, such a huge difference between avast home and pro.
     
  4. jrmhng

    jrmhng Registered Member

    Joined:
    Nov 4, 2007
    Posts:
    1,268
    Location:
    Australia
    A: OMG MY AV ROX AND YOUR AV SUXORS
    B: NO WAY THEY TEST CRAP AND EVERYTHING IS FULLY BIASED

    Just thought I could get the fan boy stuff out of the way before we start having a real discussion :p

    It is such a huge discrepancy and I cant even explain it. The last time I looked Avast wasn't doing so bad so I thought it must have been a recent 'hit' they took. But I looked at the yearly stats and they are doing just as bad but I clearly remember they were never ranked so low. Maybe there is a mistake?
     
  5. Taliscicero

    Taliscicero Registered Member

    Joined:
    Feb 7, 2008
    Posts:
    1,439
    Clam-AV is where it should most likely be, but avast is under it, so i am confused with this, by my understanding Avast and Avast pro use almost exactly the same engine.
     
  6. Sputnik

    Sputnik Registered Member

    Joined:
    Feb 24, 2005
    Posts:
    1,198
    Location:
    Москва
    In terms of detection avast! Home and Professional are exactly the same, I don't believe the results...
     
  7. Saraceno

    Saraceno Registered Member

    Joined:
    Mar 24, 2008
    Posts:
    2,404
    Regarding avast!, see this thread:
    http://forum.avast.com/index.php?topic=36109.0

    Moderator states:
    -----

    So if the detection capabilities are the same, avast! home should rank the same as the professional product. Shadowserver however, are using the linux versions of the product (which aren't as high in detection rates). See: http://www.shadowserver.org/wiki/pmwiki.php?n=Stats.Viruses

    Also, Shadowserver are using the avast! linux home edition version 1.0.8, not the latest version 4 or the windows version 4.8 most are using.

    The Kasperksy version shadowserver is using, is Kaspersky Anti-Virus for File Server, version 5.5.18.
     
    Last edited: Jun 14, 2008
  8. emperordarius

    emperordarius Registered Member

    Joined:
    Apr 27, 2008
    Posts:
    1,218
    Location:
    Who cares
    I don't believe those results. Different from any possible experience I had and any av-comparatives/test I 've seen.
     
    Last edited: Jun 14, 2008
  9. dawgg

    dawgg Registered Member

    Joined:
    Jun 18, 2006
    Posts:
    817
    Personally, I've always disliked ShadowServer's statistics and think they have always been a very misleading representation of detection rates of AVs... my opinion still stands :)
     
  10. jrmhng

    jrmhng Registered Member

    Joined:
    Nov 4, 2007
    Posts:
    1,268
    Location:
    Australia
    Why are they misleading? What is wrong with how they are doing it or how they are reporting it?
     
  11. Blackcat

    Blackcat Registered Member

    Joined:
    Nov 22, 2002
    Posts:
    4,010
    Location:
    Christchurch, UK
    I would take their results with a pinch of salt.

    For example, VirusBlokAda have had a long battle with them in trying to get them to test VBA32 fairly. They were testing VBA in demo mode as their license had expired!!!!!! Recent results suggest that maybe, just maybe they now know how to use VBA32 (and other AVs)!

    And overall Honeypots are well-known to contain a high proportion of corrupted samples.
    I agree; the use of Linux versions and honeypots do not help but it is also their methodology that I question.
     
  12. Bunkhouse Buck

    Bunkhouse Buck Registered Member

    Joined:
    May 29, 2007
    Posts:
    1,056
    Location:
    Las Vegas
    Good question.

    It may be more of the mantra here that if your AV tested well the methods are scientific and statistically valid... if it did not test well, the methodology is obviously flawed...actually I have no idea if the results are valid or not.
     
    Last edited: Jun 14, 2008
  13. Baz_kasp

    Baz_kasp Registered Member

    Joined:
    May 1, 2008
    Posts:
    593
    Location:
    London
    What put me off shadowserver is the fact that certain products....using the same engine...were getting different results for the same detection. Now if that is not an indication that something is amiss I'm not sure what is ;)
     
  14. dawgg

    dawgg Registered Member

    Joined:
    Jun 18, 2006
    Posts:
    817
    Exactly... if there was a consistent setup of AVs, certain AVs which share the same engine such as Kaspersky and F-secure will have the same number of detections. In ShaddowServer's case, a signature detection (for example of Trojan.Win32.VB.bzr), Kaspersky gets 32,000+, whereas F-Secure gets a little below 25,000... if its the same signature, why is there a discrepancy; must be from inconsistant settings.

    Also, as Blackcat rightfully mentioned, the use of honeypots results in many corrupted samples (samples which do not and can not infect you) and honeypots do not necessarily catch samples which normal users may be infected by. Also tests are carried out using Linux versions of AVs, whereby results can vary against Windows versions (which most readers use) due to ability to unpack etc.
     
  15. dawgg

    dawgg Registered Member

    Joined:
    Jun 18, 2006
    Posts:
    817
    Not with me... We all know the AV industry is a competitive and dynamic one and its not easy to be the best because all the other AVs are trying to reach the same goal... everyone cant be the best! The AV I use does not reach 100% or the highest in all tests, but I dont mind, I know it doesn't detect 100% of samples and I know which AVs detect more than mine... if another AV detects well, good for it :)
     
  16. trjam

    trjam Registered Member

    Joined:
    Aug 18, 2006
    Posts:
    9,057
    Location:
    North Carolina
    cant argue your thoughts dawgg.;)
     
  17. MalwareDie

    MalwareDie Registered Member

    Joined:
    Dec 8, 2006
    Posts:
    500
    They don't share ths same engine. F-Secure uses the old Kaspersky engine.
     
  18. plantextract

    plantextract Registered Member

    Joined:
    Feb 13, 2007
    Posts:
    392
    they are both linux versions, so kav for linux is still 5.5, there's no heuristics engine like in the windows version 7 and so on.
     
  19. dawgg

    dawgg Registered Member

    Joined:
    Jun 18, 2006
    Posts:
    817
    Detection rates should still be the same between the versions used in ShadowServer (because it uses a Kaspersky version which has the old engine, which F-Secure also uses). Signature detection of Kaspersky should be the same as F-secure in ShadowServer, the only detection difference in the versions ShadowServer uses should be that F-Secure also has its other engines, but in the results, F-Secure's engines dont detect anything (possibly another flaw of using older linux versions; current Windows version of F-Secure may be able to detect more using Heuristics). In this case, there is no evidence any of the heuristics were used, so F-Secure and Kaspersky are only getting detections using Kaspersky's engine, so the results should be the same. Kaspersky didnt have its Heuristic module in the version ShadowServer uses which is why the results are partially flawed; in reality, most users will have heuristics detecting malware as well in Kaspersky).

    Other AV products will also most likely have discrepancies as such and have current windows versions of their products detecting far more than ShadowServer does. Eg, DrWeb's Windows home user products uses the Orion heuristics... ShadowServer doesn't, hence in the real world, Orion may detect malware, but ShadowServer will show it does not detect it.



    I am not attacking ShadowServer and it is good that it is transparent with its setup and information about its methodology, but the problem is that many people only look at the numbers and know absolutely nothing about the outdated versions compared, samples (many of which are likely corrupted) and OS; hence in the real world, its not showing a proper up to date comparison of detection capabilities; hence IMO, ShadowServer's results should not be considered when comparing detection capabilities of AVs for home-users (most of which are Windows users).
    Also, seeing as it shows one signature is capable of detecting far over 50% of the malware detected that day, it does imply the samples used are questionable to say the least.
     
  20. Diver

    Diver Registered Member

    Joined:
    Feb 6, 2005
    Posts:
    1,444
    Location:
    Deep Underwater
    I always wonder why Symantec is not represented. Do they not make a Linux version or is it too expensive?
     
  21. Macstorm

    Macstorm Registered Member

    Joined:
    Mar 7, 2005
    Posts:
    2,531
    Location:
    Sneffels volcano
    norman, f-prot, avast, avg, nod32, mcafee & vba32...all of them above kav...oh well :rolleyes:
     
  22. AndreyKa

    AndreyKa Registered Member

    Joined:
    Feb 25, 2005
    Posts:
    93
    Location:
    Russia
    IMHO there is no way to disable origin detection in 4.44 engine.
    So, ShawdowServer still use 4.33 version?
     
  23. trjam

    trjam Registered Member

    Joined:
    Aug 18, 2006
    Posts:
    9,057
    Location:
    North Carolina
    this site deals with zero day threats, so if you look hard enough, there is a lot of reality to their findings.
     
  24. Bunkhouse Buck

    Bunkhouse Buck Registered Member

    Joined:
    May 29, 2007
    Posts:
    1,056
    Location:
    Las Vegas
    But that "reality" is not acceptable to many of the true believers whose AV is in the lower end of the rankings- including Kaspersky, etc.
     
  25. Baz_kasp

    Baz_kasp Registered Member

    Joined:
    May 1, 2008
    Posts:
    593
    Location:
    London

    No, that "reality" is not acceptable when F secure is getting twice as many detections as KAV using a Kaspersky detection from the Kaspersky engine.


    That and the fact that most honeypots are filled to the gills with corrupted samples that most vendors will not add to their detection lists. I don't see any methodology there that states they check for corrupted samples, the frequency of updates and what settings each engine is tweaked to.
     
Loading...
Thread Status:
Not open for further replies.