Personal AV tests

Discussion in 'other anti-virus software' started by dan_maran, Apr 30, 2005.

Thread Status:
Not open for further replies.
  1. dan_maran

    dan_maran Registered Member

    Joined:
    Aug 30, 2004
    Posts:
    1,053
    Location:
    Stamford, CT
    As of late, it seems people have been shying away from posting results of personal testing scenarios. I find this disturbing, because in my opinion that seems like they are just taking others "advice" and not trying things out for themselves. Or I could be completely wrong, and way off the mark.

    Well, who knows. Any how I have been testing a few AV's against my personal collection(s) and have posted the results, along with the lame test bed log file. You can view them here

    But please read the Disclaimer and form your own opinions. As this is currently a work in progress, it is fairly lame. You also need some understanding of log files to grasp the information provided. But please remember, as you will see in the Disclaimer, I am no Professional and this is merely a test for myself I felt others might enjoy.
    And if you don't read the Disclaimer , then don't even bother posting a reply to this thread.

    Also, if you feel something should be changed, or you need some kind of more information please say so.

    //01MAY05//
    I had to move the site so the links are updated now
    http://oem.zer0-tec.net/testing/DISCLAIMER.html
     
    Last edited: May 1, 2005
  2. christophs

    christophs Registered Member

    Joined:
    Oct 12, 2004
    Posts:
    23
    Thanks!
    Are you only testing two AVs?
     
  3. Firecat

    Firecat Registered Member

    Joined:
    Jan 2, 2005
    Posts:
    7,927
    Location:
    The land of no identity :D
    I'm sure there's more coming soon :)
     
  4. Happy Bytes

    Happy Bytes Guest

    I've absolutely no doubts about this :blink:
     
  5. Blackcat

    Blackcat Registered Member

    Joined:
    Nov 22, 2002
    Posts:
    4,010
    Location:
    Christchurch, UK
    I enjoy seeing personal tests here, as long as were not talking about sample sizes of 10-100.

    likuidkewl, my only suggestion would be that you try to count the number of undetected files after the scanner has been set to delete infected samples.

    This will reduce the time spent looking through the log-files and may be more accurate, as in some situations such as packed malware, some AV scanners count the same sample several times.

    Just copy your collection to CD/flashdrive and then use the delete.

    Keep testing ;)

    What proportion of your test-beds, particularly your large collection of over 57,000 samples, were DOS viruses?
     
    Last edited: Apr 30, 2005
  6. IBK

    IBK AV Expert

    Joined:
    Dec 22, 2003
    Posts:
    1,819
    Location:
    Innsbruck (Austria)
    could you please if possible also provide a CRC32 list of the used files? if u are interested to get from me a feedback about the samples based on the crc32 lists just let me know by PM.
     
  7. Dorelian

    Dorelian Registered Member

    Joined:
    Apr 27, 2005
    Posts:
    1
    No offence or anything, but personally, I find this very inaccurate.

    First of all, your samples aren't unique. Some examples:
    E:\Live\ANARKIA1.COM : infected Jerusalem.Curse.1653.b
    E:\Live\ANARKIA2.COM : infected Jerusalem.Curse.1653.b
    E:\Live\ANARKIA2.EXE : infected Jerusalem.Curse.1653.b

    E:\Live\B-560P.COM : infected Burger.560.z
    E:\Live\B-560Q.COM : infected Burger.560.z
    E:\Live\B-560S.COM : infected Burger.560.z
    E:\Live\B-560V.COM : infected Burger.560.z
    E:\Live\B-560W.COM : infected Burger.560.z

    E:\Live\BAMMPC1.COM : infected PS-MPC-based.a
    E:\Live\BAMMPC10.COM : infected PS-MPC-based.a
    E:\Live\BAMMPC2.COM : infected PS-MPC-based.a
    E:\Live\BAMMPC3.COM : infected PS-MPC-based.a
    E:\Live\BAMMPC4.COM : infected PS-MPC-based.a
    E:\Live\BAMMPC5.COM : infected PS-MPC-based.a
    E:\Live\BAMMPC6.COM : infected PS-MPC-based.a
    E:\Live\BAMMPC7.COM : infected PS-MPC-based.a
    E:\Live\BAMMPC8.COM : infected PS-MPC-based.a
    E:\Live\BAMMPC9.COM : infected PS-MPC-based.a
    E:\Live\BAMST-01.COM : infected PS-MPC-based.a
    E:\Live\BAMST-02.COM : infected PS-MPC-based.a
    E:\Live\BAMST-03.COM : infected PS-MPC-based.a
    E:\Live\BAMST-04.COM : infected PS-MPC-based.a
    E:\Live\BAMST-05.COM : infected PS-MPC-based.a
    E:\Live\BAMST-06.COM : infected PS-MPC-based.a
    E:\Live\BAMST-07.COM : infected PS-MPC-based.a
    E:\Live\BAMST-08.COM : infected PS-MPC-based.a

    You say you have a collection of 4112 viruses. How much of those are actually unique, and not duplicates ? The TWO(!) antiviruses you tested both detected around 95%(94% and 96%, to be exact). How am I supposed to know they didn't both detect 100% ? How do I know the remaining 5% are actually viruses, since there's no list of them ALL anywhere ?

    Second, you didn't mention what settings you ran the antiviruses on.

    I'm sorry, but I find this test highly inaccurate, and it doesn't in any way alter my opinion about any of those two products. Despite your disclaimer, you're just confusing less experienced people with this test. :/
     
  8. Happy Bytes

    Happy Bytes Guest

    When i see THIS FILENAMES i already know from which Zip file it was downloaded from the internet! It was together with 3500 other old dos viruses. And nearly 35% of the samples out of this 3500 are DEAD! I know that because i did analyse this samples! This collection does also include wrong cleaned dos viruses - McAfee was some years back the master of turning corrupting samples during cleaning - such files are in this collection! Even with the tag inside the files from NORTON ANTIVIRUS that it did clean it!
     
  9. Happy Bytes

    Happy Bytes Guest

    And with this i prove to you what i just said:

    OUT OF YOUR OWN SCANLOGS !!! And this is only a part! I didn't quote here all samples! Beside of this there is MORE trash which Norman doesnt flag as garbage!

    This are all WRONG CLEANED samples from McAfee and Norton! But nobody believes me because i have to be evil to disagree with such tests :rolleyes:
    EOT.
     
  10. Happy Bytes

    Happy Bytes Guest

    Every morning when i get up from the bed i spend atleast 10 minits in praying that only people which know how to calculate a MS DOS Executable Entrypoint only with the help of a hexeditor should do antivirus tests... :rolleyes:

    ...doesn't work :eek:

    That said: AV tests are not a game - the tester needs experience in this field - not only how to use a antivirus program! Normally he should be able to verify his own samples with a disassembly without the need of other scanners!
     
  11. RejZoR

    RejZoR Registered Member

    Joined:
    May 31, 2004
    Posts:
    6,426
    But on the other hand,checking of 30k samples 1 by 1 is no fun ;)
     
  12. dan_maran

    dan_maran Registered Member

    Joined:
    Aug 30, 2004
    Posts:
    1,053
    Location:
    Stamford, CT
    Wow, nothing like waking up seeing people just straight trashing something you said from the begining was not scientific nor accurate. But a personal test, that you would like to share with others.

    As for the constructive comments, thanks and I will try what was suggested.

    *Removed*

    Good day!
     
    Last edited: Apr 30, 2005
  13. Happy Bytes

    Happy Bytes Guest

    You have to REPLICATE virus samples before you do such tests!

    Otherwise every scanner can CHEAT! They just need to add a CRC for difficult to detect polymorphic viruses out of such test sets! And the result
    out of this is that the scanner might fail to detect a REAL WORKING virus in another file!

    There is a very simple rule - replicate 5 generations of file infectors, and take the 3rd generation for you test! (cause you know then that this files are working because they did infect already 2 other generation samples) IS THIS REALLY TO MUCH EXPECTED TO UNDERSTAND THIS SIMPLE PROCEDURE FOR TESTING FILEINFECTOR VIRUSES WHICH ARE NON-METHAMORPHIC o_O?

    What do you expect from helping? That i teach you disassembly?
     
  14. dan_maran

    dan_maran Registered Member

    Joined:
    Aug 30, 2004
    Posts:
    1,053
    Location:
    Stamford, CT
    Thanks for that information.

     
  15. Happy Bytes

    Happy Bytes Guest

    If there would be a REAL interest i would even give a workshop here how to do this in a proper way, but i'm afraid (and this is not personally against you) that most of the people wouldn't understand the in-depth procedures.

    It's very complex, and not only done with collecting a few samples. If the community here is interested then reply in this thread - otherwise don't shut at me when i say it's a amateurish test. It's easy as this. At least 50 interested people are needed, otherwise it makes no sense to spend so much time into details. Up to the readers here :-*
     
  16. IBK

    IBK AV Expert

    Joined:
    Dec 22, 2003
    Posts:
    1,819
    Location:
    Innsbruck (Austria)
    I proposed to help likuidkewl and FireFighter to sort out the garbage from their collection by sending my CRC32 logs (as I guess they would not like to send me the samples); but so far I did not got such logs neither from FireFighter (which I asked already last week) nor from likuidkewl (which I asked some hours ago). My question is: why you do not let you help a little bit to sort out known garbage files to improve a bit your sets?
     
  17. dan_maran

    dan_maran Registered Member

    Joined:
    Aug 30, 2004
    Posts:
    1,053
    Location:
    Stamford, CT
    I will work on that some more. I do want some help weeding them out, that was a big reason I posted it here at Wilders. Also, IBK, I am in a completely different time zone and you were all (well mostly) posting while I was in bed.
    Also as I am not a Professional, as previously stated, I have other things to attend to before I can start again. Mainly a wife ;)
     
  18. IBK

    IBK AV Expert

    Joined:
    Dec 22, 2003
    Posts:
    1,819
    Location:
    Innsbruck (Austria)
    Ok, good :), just let me know when ready.
    Now I am waiting for FireFighter CRC32 logs ;)
     
  19. mikel108

    mikel108 Registered Member

    Joined:
    Dec 10, 2004
    Posts:
    1,057
    Location:
    SW Ontario, Canada
    Which is why I think he put a disclaimer on the site. I am sure that I will see your testing up soon. Thanks
     
  20. mikel108

    mikel108 Registered Member

    Joined:
    Dec 10, 2004
    Posts:
    1,057
    Location:
    SW Ontario, Canada

    With respect Happy Bytes. I find you very knowledgeable. I was wondering if you have done any testing of your own and what the results have been??

    Thanks
     
  21. Happy Bytes

    Happy Bytes Guest

    I analyse complex viruses since years, including polymorphic & methamorphic there is no need for me to prove this with a own antivirus test. Beside of this it would be unfair, cause i work for ESET.
     
  22. dan_maran

    dan_maran Registered Member

    Joined:
    Aug 30, 2004
    Posts:
    1,053
    Location:
    Stamford, CT
    Files are in the process of being weeded out, new scans will be up shortly.
    Thank you for your inputs.
    *Logs will now be organized by CRC32 with the same information as before*
     
  23. IBK

    IBK AV Expert

    Joined:
    Dec 22, 2003
    Posts:
    1,819
    Location:
    Innsbruck (Austria)
    I got the CRC32 logs from likuidkewl and I sent him a list of files that should be removed as they are known garbage. The list is huge but not complete, anyway atm I can not do more without having the samples in front of me. HappyBytes was right when he said that quite much garbage is contained. likuidkewl please check your mailbox and remove the reported files, after that I will try to check more in depth the crc32 logs in order identify more known garbage ;-) in order to improve your set.
     
  24. Happy Bytes

    Happy Bytes Guest

    :rolleyes: :rolleyes: :rolleyes: :rolleyes: :rolleyes: :rolleyes: :rolleyes: :rolleyes: :rolleyes: :rolleyes:
     
  25. dan_maran

    dan_maran Registered Member

    Joined:
    Aug 30, 2004
    Posts:
    1,053
    Location:
    Stamford, CT
    Thank you. This will help others out also.
    Very constructive NOT destructive! :rolleyes:

    Just so everyone knows, 34 (0.1%) files were removed from small collection, 3299 is the new number.
    Large is still getting groomed.
     
    Last edited: Apr 30, 2005
Loading...
Thread Status:
Not open for further replies.