AV-Comparatives Anti Trojan Test ...

Discussion in 'other anti-trojan software' started by Infinity, Mar 24, 2006.

Thread Status:
Not open for further replies.
  1. SSK

    SSK Registered Member

    Joined:
    Nov 28, 2004
    Posts:
    976
    Location:
    Amsterdam
    Interesting comparison. Thanks Firefighter! :)
     
  2. Firefighter

    Firefighter Registered Member

    Joined:
    Oct 28, 2002
    Posts:
    1,670
    Location:
    Finland
    I posted that only because VirusP tests have very poor reputation in here. Maybe, some av-programs were not performed so well before in VirusP, but nowadays they are performing pretty well in VirusP too. Anyway, those results are only those that matters.


    Best regards,
    Firefighter!
     
  3. controler

    controler Guest

    Can I ask why a AT has to have a file scanner or did I miss that part?
    The encrypted file has to decrypt to run doesn't it?
    I think boClean can scan a file by dragging and dropping into GUI.

    Also if BoClean was tested a year ago , they now have a low level driver making it even better. You can't even find BoClean mentioned on the Holyfather site.
    Second, it is not the quainty of sig but rather the quality, right?
    Any program that starts doing it's thang before logon is cool.

    My best advice is for users to download their own crapware and just go ahead and try it on their scanners and then decide for themselves.
    there are more and more doing it now days then a few years ago.

    Do not listen to anybody else, just do it!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

    Then make your own judgements.

    Winhex, Morphine ect are all out there to be downloaded.
    scamble & bake them anyway you want and then go to run them.

    Most all new stuff can easliy be picked up P2P filesharing anyways. Gee I wonder how many sig writers go there? LOL ALL?

    Every hijackthis log fixerupper person says hey go download Ewido. Why? well because you can get a free version of BoClean unless you want to buy it for 30 days, clean your system, then get your money refunded.
    Ok I guess it is easier to do. I agree.

    con
     
  4. Notok

    Notok Registered Member

    Joined:
    May 28, 2004
    Posts:
    2,969
    Location:
    Portland, OR (USA)
    I can't imagine running 150,000+ files one at a time, 9 times over, and recording the results for each. I also can't imagine individually unpacking that many samples, it would just be too much.

    I don't think there was much invested in the scanner, it probably wouldn't be able to detect anything that's compressed or crypted in any way.

    Possibly because holy_father quit

    Right, so you can get rediculous results like VirusP's tests. I'd rather see accurate results. Those collections are too full of stuff that shouldn't be detected.. leave it to the folks that can tell the difference before running the test.
     
  5. Detox

    Detox Retired Moderator

    Joined:
    Feb 9, 2002
    Posts:
    8,507
    Location:
    Texas, USA
    I think that would be rather irresponsible advice; especially when given to the average PC user.
     
  6. siliconman01

    siliconman01 Registered Member

    Joined:
    Mar 6, 2003
    Posts:
    786
    Location:
    West Virginia (USA)
    The issue that bothers me the most is that even the program that scored highest on this test allowed 23% of the malicious items to get past it. That is highly unacceptable in my book. I feel that anything short of 99% detection/removal rate is "bad news". I certainly would not accept an automobile that has a failure rate of 23%! Why do we seem willing to accept this type of performance from software that is suppose to protect us against criminal activity that steals our money and our identity?

    It seems rational to me that if a testing facility can install malicious elements on test computers that the malicious elements are KNOWN. Therefore there is no excuse for why it is not detected by malware detection software that is marketed. We should be pounding on the doors of all such software vendors who have accepted our money- demanding that they correct their shortcomings. There is no reason in my opinion to accept otherwise.

    That being said, I honestly don't blame any software vendor who desires to not participate in this type of testing. To the best of my knowledge, there are no published and generally accepted international standards/criteria for testing this type of software. Nor is there a published list of certified, independently audited, testing facilities...the list showing who which ones are in compliance with the "non-existent" testing standards. This lack of standards and compliance requirements makes any such test questionable at best.

    Nor is there any rationale for any of the vendors to claim superiority when detection rates are less than 99% of whatever is known to be out there and whatever it is that they are suppose to protect against.

    Plus I feel testers who have all these "undetected" malicious items should immediately submit all of it to the various security software vendors so that these vendors can promptly add it to their rulesets/definitions/whatever.

    JMO ;)
     
    Last edited: Mar 25, 2006
  7. Blackcat

    Blackcat Registered Member

    Joined:
    Nov 22, 2002
    Posts:
    4,024
    Location:
    Christchurch, UK
    Spoken like a dedicated TrojanHunter Forum helper ;)

    Granted that there are more difficulties in testing trojan rather than virus/worm detection but as yet no one has questioned the trojan detection of the AV programs over at av-comparatives.

    No discussion yet over at the TH forums. Would be good if Magnus or Gavin gave their view of the testing procedure used and why they were not happy with the results.

    It is becoming harder and harder to decide which test sites give results which can be relied upon to provide some reliable data to support security software choice!
     
  8. IBK

    IBK AV Expert

    Joined:
    Dec 22, 2003
    Posts:
    1,886
    Location:
    Innsbruck (Austria)
    btw, a tester should not be a source to get malware.
    anyway, like written in the report, even the ATs with low on-demand detection rates can make sense.
     
  9. siliconman01

    siliconman01 Registered Member

    Joined:
    Mar 6, 2003
    Posts:
    786
    Location:
    West Virginia (USA)
    Guess I do not understand why not. Testers of MS Windows promptly submit flaws and security holes that they find to MS. Testers of FireFox do the same for this vendor.

    I thought the primary objective is to protect computer users against criminal and/or malicious activity, irrespective of who and how is it discovered. Why play Hide and Seek with security software vendors...most of whom are busting their buns to stay ahead of the bad guys. :(
     
  10. IBK

    IBK AV Expert

    Joined:
    Dec 22, 2003
    Posts:
    1,886
    Location:
    Innsbruck (Austria)
    thats a completly different thing.
    anyway i am not open to discuss about this. If a producer has not enough sources of malware to protect its customers agisnt malware, then its product is probably obsolete and useless. If tomorow xy makes an AT which detects 10 samples and after years still only 100 samples, and a tester confirms its low detection rates compared to other AT that also did not get any sample from the tester, I see no reason why a tester should send hundreds of tousands of samples to that vendor.
     
  11. rdsu

    rdsu Registered Member

    Joined:
    Jun 28, 2003
    Posts:
    4,537
    IBK,

    If a developer let you test its product, why should you give them the option to not publish their results!?

    For me, this doesn't make any sense and you should avoid this kind of things in future...

    As I already aspect, ewido is the best on this area! :D

    Thanks for this test :)
     
  12. IBK

    IBK AV Expert

    Joined:
    Dec 22, 2003
    Posts:
    1,886
    Location:
    Innsbruck (Austria)
    @VaMPiRiC_CRoW: the things is, I did not asked to the AT vendors to send me a written permission (like I do with the AV vendors), so I have only the post in this forum and the mail where he allows to test. But I do not have a written permission from TH to publish the results, and as in Austria the media laws are very strict, why should I look for troubles by publishing for free TH results if TH prefers to be not listed? That was the simplest and fastest option. I am not interested to test an AT product in my free time and then to get biased or insulted from an AT vendor just because he is unhappy with the test and results. I hope this is understandable.
    What makes me a bit unhappy is that it seems like the main interest is in TH which did not wna to be listed, instead of the other things written in the report :/. One week for the AT test report spent and not much interest in what it contains... :(
     
  13. Happy Bytes

    Happy Bytes Guest

    They did insult you? :eek:
     
  14. rdsu

    rdsu Registered Member

    Joined:
    Jun 28, 2003
    Posts:
    4,537
    I understand why you did that :)

    Regards
     
  15. Acadia

    Acadia Registered Member

    Joined:
    Sep 8, 2002
    Posts:
    4,325
    Location:
    US
    IBK, I'm interested. :thumb:

    Acadia
     
  16. StevieO

    StevieO Registered Member

    Joined:
    Feb 2, 2006
    Posts:
    1,067
    IBK

    Of course we are all Very interested, and grateful too ! It's just that the TH episode has understandably raised a lot of questions.

    I don't see why you need to let ANY of them know in advance, or ask them for permision etc when you do the tests, or to publish the results etc !


    StevieO
     
  17. IBK

    IBK AV Expert

    Joined:
    Dec 22, 2003
    Posts:
    1,886
    Location:
    Innsbruck (Austria)
    well, tell it to those who makes the media laws in austria :p
    i think i explained quite well the reasons.
     
  18. controler

    controler Guest

    I think IBK did a great job testing and will stand up and applaud him. At least he is trying.
    Again , same as above. I was not advising average PC users to do it. I was targeting the mainstream security forum posters that like to post good or bad things about a product based on what other experts have written.
    I know allot of times my post are not clear enough but then again I hated grammar in school. Other love to write long explanatory posts and I love to read them all too.

    Oh and yes I have been bashed many times here in the past and accused of scare mongering for saying things like ROOT KIT
    or worse yet, residual memory ROOT KIT. I did not say these thing because this was some new technology coming maybe 10 years from now as most said. IT WAS HERE NOW. I think some need to now go back to those threads and eat a little crow, don't you?
    I do not know if I see much linking to the root kit and black hat site, but I do see it allot over at DSLR.
    Linking directly to the same sites not allowed here. TRUE info on both sites is proof of concept but still is not allowed here.
    Maybe for the average user as Detox puts it, the knowledge would be too overwhelming at those sites but hey guess what?
    allot of good & bad people gain knowledge at those sites. I am sure many gov agencies frequent them also.

    The I really like is the fact that most of the renowned authors on those sites now not only provide proof of concept attacks but also talk about ways of detection. Then again most of the regulars here and on DSLR frequent those sites now and gain a wealth of info, whereas a few years ago they didn't think it was right or should I say moral to even visit a black hat or root kit site.

    Oh dear, sorry for my rambling again. If you got this far you deserve a cup a joe :)

    As a appendix to the statement on testing yourself. Find the newest and also repack and maybe even make some new for your own testing pleasure and not to be used in any other manner.


    con
     
  19. Infinity

    Infinity Registered Member

    Joined:
    May 31, 2004
    Posts:
    2,651
    :D
    Those are my beliefs too C.

    I hated grammar too @ school .. I was not bashing TH based on what experts have written. - I still hope we are clear about this!

    We all know that members @ security section @ DSLR are of other caliber then the average member @ Wilders .. funny you mentioning it btw ..

    that's my understanding too. it would be unheard off gov agencies not frequenting Security fora.

    Again, I do applaud for every - somewhat experienced - user to visit blackhat/whitehat/rootkit site to gain info, to go into detail about hooking/kernel driven software/drivers, code injection and permutations .. all kinds ....

    It is rediculous to state that those members/visitors are only there to do "bad" things.
    I do believe there is some kind of science in this all, mathematics, codes, ... and to go "into the field" can sometimes be more satisfying with better results then the actual "theory".

    Have a good weekend too and when you're testing, be sure to get your settings straight on your VmWare console .. or you'll infect da damn thing :D:D
     
  20. Notok

    Notok Registered Member

    Joined:
    May 28, 2004
    Posts:
    2,969
    Location:
    Portland, OR (USA)
    Great report, IBK, and thanks. Tests of apps other than AVs is something that's been needed for some time, and I hope you will continue to do them.

    If/when you do the next one, could you include Antiy Ghostbusters? That's one that crops up every once in a while, and I know I've been a little curious about it, I'm sure others are as well.
     
  21. Gavin - DiamondCS

    Gavin - DiamondCS Former DCS Moderator

    Joined:
    Feb 10, 2002
    Posts:
    2,080
    Location:
    Perth, Western Australia
    Imperfect testing is likely, a few things in TrojanHunter's defense, since noone else knows the facts. Unpacked samples are NOT true ITW samples for sakes of a scanner which does not CHOOSE to be an AV scanner with certain design rules and strategies. TH falls in this category as do some others.

    It also disadvantages TH if sample files themselves are not named in a specific way. It cares about REAL LIVE MALWARE, not test sets with .$XE extensions or .VIR or anything else which will not run and is just sitting in some hard drive ! this is a simple design decision to speed up scanning in TrojanHunter.

    This mis detection and others will be addressed in future versions of TrojanHunter so that future "tests" are not so unfavourable at first look. If YOU the users care so much about them.. keep in mind working JUST to satisfy a test wastes REAL analysis and program development time. TDS-3 is now dead but it did GREAT in tests, which I find irony at it's best in this case. Tests are not the be-all and end all unless they are concise and completely OPEN. We were not allowed even one sample or any information on what the testset contains. NO information.

    And for samples which are OLD, these are nearly always not used. It would be best to gather huge amounts of ITW samples over time and test them with a 6 months backlog for compensation. Once files are no longer seen ITW they are removed from the testset. In memory testing, cleaning testing and more. How long would this take the testers ? months on end.. years..
     
  22. IBK

    IBK AV Expert

    Joined:
    Dec 22, 2003
    Posts:
    1,886
    Location:
    Innsbruck (Austria)
    of course all files have executable extensions, like I already said to Magnus.
    What the test does show is described in the report and also why the rates may look low.
    How the test was going to be done was known, so if TH had problems with that, why it wasn't said in first place?
     
    Last edited: Mar 26, 2006
  23. Happy Bytes

    Happy Bytes Guest

    Code:
    HKEY_CLASSES_ROOT\.dli
    HKEY_LOCAL_MACHINE\software\classes\.dli
    
    ---> Kernel32.DLi [Netdevil 1.5] so basically it makes more sense if you use fixed scan extensions to have a mask like ".EX*", ".DL*" ".OC*", ".CO*" etc.

    However, it is of course true that you have to name samples with the correct extension for proper testresults. Most of the AV Programs having for instance problems with wrong extensions for Bootsector Image Files. Such files are not really executable from the system via double-click or command line, they just reflect a boot sector image in order to test for bootsector viruses.
     
  24. Gavin - DiamondCS

    Gavin - DiamondCS Former DCS Moderator

    Joined:
    Feb 10, 2002
    Posts:
    2,080
    Location:
    Perth, Western Australia
    I noticed on the site you do acknowledge imperfect testing, and that the amount of hours to do perfect testing makes it totally out of the question. I'd rather help perfect testing than spend time helping a test see the product detecting malware.
     
  25. IBK

    IBK AV Expert

    Joined:
    Dec 22, 2003
    Posts:
    1,886
    Location:
    Innsbruck (Austria)
    you should spend your time in improving your product and acknowledge its imperfection, instead of trying to bash a test due the results of your product.
     
Thread Status:
Not open for further replies.
  1. This site uses cookies to help personalise content, tailor your experience and to keep you logged in if you register.
    By continuing to use this site, you are consenting to our use of cookies.