Personal AV tests

As of late, it seems people have been shying away from posting results of personal testing scenarios. I find this disturbing, because in my opinion that seems like they are just taking others "advice" and not trying things out for themselves. Or I could be completely wrong, and way off the mark.

Well, who knows. Any how I have been testing a few AV's against my personal collection(s) and have posted the results, along with the lame test bed log file. You can view them here

But please read the Disclaimer and form your own opinions. As this is currently a work in progress, it is fairly lame. You also need some understanding of log files to grasp the information provided. But please remember, as you will see in the Disclaimer, I am no Professional and this is merely a test for myself I felt others might enjoy.
And if you don't read the Disclaimer , then don't even bother posting a reply to this thread.

Also, if you feel something should be changed, or you need some kind of more information please say so.

//01MAY05//
I had to move the site so the links are updated now
http://oem.zer0-tec.net/testing/DISCLAIMER.html

 

Thanks!
Are you only testing two AVs?

 

I'm sure there's more coming soon

 

I've absolutely no doubts about this

 

I enjoy seeing personal tests here, as long as were not talking about sample sizes of 10-100.

likuidkewl, my only suggestion would be that you try to count the number of undetected files after the scanner has been set to delete infected samples.

This will reduce the time spent looking through the log-files and may be more accurate, as in some situations such as packed malware, some AV scanners count the same sample several times.

Just copy your collection to CD/flashdrive and then use the delete.

Keep testing

What proportion of your test-beds, particularly your large collection of over 57,000 samples, were DOS viruses?

 

could you please if possible also provide a CRC32 list of the used files? if u are interested to get from me a feedback about the samples based on the crc32 lists just let me know by PM.

 

No offence or anything, but personally, I find this very inaccurate.

First of all, your samples aren't unique. Some examples:
E:\Live\ANARKIA1.COM : infected Jerusalem.Curse.1653.b
E:\Live\ANARKIA2.COM : infected Jerusalem.Curse.1653.b
E:\Live\ANARKIA2.EXE : infected Jerusalem.Curse.1653.b

E:\Live\B-560P.COM : infected Burger.560.z
E:\Live\B-560Q.COM : infected Burger.560.z
E:\Live\B-560S.COM : infected Burger.560.z
E:\Live\B-560V.COM : infected Burger.560.z
E:\Live\B-560W.COM : infected Burger.560.z

E:\Live\BAMMPC1.COM : infected PS-MPC-based.a
E:\Live\BAMMPC10.COM : infected PS-MPC-based.a
E:\Live\BAMMPC2.COM : infected PS-MPC-based.a
E:\Live\BAMMPC3.COM : infected PS-MPC-based.a
E:\Live\BAMMPC4.COM : infected PS-MPC-based.a
E:\Live\BAMMPC5.COM : infected PS-MPC-based.a
E:\Live\BAMMPC6.COM : infected PS-MPC-based.a
E:\Live\BAMMPC7.COM : infected PS-MPC-based.a
E:\Live\BAMMPC8.COM : infected PS-MPC-based.a
E:\Live\BAMMPC9.COM : infected PS-MPC-based.a
E:\Live\BAMST-01.COM : infected PS-MPC-based.a
E:\Live\BAMST-02.COM : infected PS-MPC-based.a
E:\Live\BAMST-03.COM : infected PS-MPC-based.a
E:\Live\BAMST-04.COM : infected PS-MPC-based.a
E:\Live\BAMST-05.COM : infected PS-MPC-based.a
E:\Live\BAMST-06.COM : infected PS-MPC-based.a
E:\Live\BAMST-07.COM : infected PS-MPC-based.a
E:\Live\BAMST-08.COM : infected PS-MPC-based.a

You say you have a collection of 4112 viruses. How much of those are actually unique, and not duplicates ? The TWO(!) antiviruses you tested both detected around 95%(94% and 96%, to be exact). How am I supposed to know they didn't both detect 100% ? How do I know the remaining 5% are actually viruses, since there's no list of them ALL anywhere ?

Second, you didn't mention what settings you ran the antiviruses on.

I'm sorry, but I find this test highly inaccurate, and it doesn't in any way alter my opinion about any of those two products. Despite your disclaimer, you're just confusing less experienced people with this test. :/

 

When i see THIS FILENAMES i already know from which Zip file it was downloaded from the internet! It was together with 3500 other old dos viruses. And nearly 35% of the samples out of this 3500 are DEAD! I know that because i did analyse this samples! This collection does also include wrong cleaned dos viruses - McAfee was some years back the master of turning corrupting samples during cleaning - such files are in this collection! Even with the tag inside the files from NORTON ANTIVIRUS that it did clean it!

 

And with this i prove to you what i just said:

OUT OF YOUR OWN SCANLOGS !!! And this is only a part! I didn't quote here all samples! Beside of this there is MORE trash which Norman doesnt flag as garbage!

This are all WRONG CLEANED samples from McAfee and Norton! But nobody believes me because i have to be evil to disagree with such tests
EOT.

 

Every morning when i get up from the bed i spend atleast 10 minits in praying that only people which know how to calculate a MS DOS Executable Entrypoint only with the help of a hexeditor should do antivirus tests...

...doesn't work

That said: AV tests are not a game - the tester needs experience in this field - not only how to use a antivirus program! Normally he should be able to verify his own samples with a disassembly without the need of other scanners!

 

But on the other hand,checking of 30k samples 1 by 1 is no fun

 

Wow, nothing like waking up seeing people just straight trashing something you said from the begining was not scientific nor accurate. But a personal test, that you would like to share with others.

As for the constructive comments, thanks and I will try what was suggested.

*Removed*

Good day!

 

You have to REPLICATE virus samples before you do such tests!

Otherwise every scanner can CHEAT! They just need to add a CRC for difficult to detect polymorphic viruses out of such test sets! And the result
out of this is that the scanner might fail to detect a REAL WORKING virus in another file!

There is a very simple rule - replicate 5 generations of file infectors, and take the 3rd generation for you test! (cause you know then that this files are working because they did infect already 2 other generation samples) IS THIS REALLY TO MUCH EXPECTED TO UNDERSTAND THIS SIMPLE PROCEDURE FOR TESTING FILEINFECTOR VIRUSES WHICH ARE NON-METHAMORPHIC ?

What do you expect from helping? That i teach you disassembly?

 

Thanks for that information.

 

If there would be a REAL interest i would even give a workshop here how to do this in a proper way, but i'm afraid (and this is not personally against you) that most of the people wouldn't understand the in-depth procedures.

It's very complex, and not only done with collecting a few samples. If the community here is interested then reply in this thread - otherwise don't shut at me when i say it's a amateurish test. It's easy as this. At least 50 interested people are needed, otherwise it makes no sense to spend so much time into details. Up to the readers here

 

I proposed to help likuidkewl and FireFighter to sort out the garbage from their collection by sending my CRC32 logs (as I guess they would not like to send me the samples); but so far I did not got such logs neither from FireFighter (which I asked already last week) nor from likuidkewl (which I asked some hours ago). My question is: why you do not let you help a little bit to sort out known garbage files to improve a bit your sets?

 

I will work on that some more. I do want some help weeding them out, that was a big reason I posted it here at Wilders. Also, IBK, I am in a completely different time zone and you were all (well mostly) posting while I was in bed.
Also as I am not a Professional, as previously stated, I have other things to attend to before I can start again. Mainly a wife

 

Ok, good , just let me know when ready.
Now I am waiting for FireFighter CRC32 logs

 

Which is why I think he put a disclaimer on the site. I am sure that I will see your testing up soon. Thanks

 

With respect Happy Bytes. I find you very knowledgeable. I was wondering if you have done any testing of your own and what the results have been??

Thanks

 

I analyse complex viruses since years, including polymorphic & methamorphic there is no need for me to prove this with a own antivirus test. Beside of this it would be unfair, cause i work for ESET.

 

Files are in the process of being weeded out, new scans will be up shortly.
Thank you for your inputs.
*Logs will now be organized by CRC32 with the same information as before*

 

I got the CRC32 logs from likuidkewl and I sent him a list of files that should be removed as they are known garbage. The list is huge but not complete, anyway atm I can not do more without having the samples in front of me. HappyBytes was right when he said that quite much garbage is contained. likuidkewl please check your mailbox and remove the reported files, after that I will try to check more in depth the crc32 logs in order identify more known garbage ;-) in order to improve your set.

 

Thank you. This will help others out also.
Very constructive NOT destructive!

Just so everyone knows, 34 (0.1%) files were removed from small collection, 3299 is the new number.
Large is still getting groomed.