Check this out....Antivirus Malware-Test Lab Dec 11, 2006

[trjam]

Posted by Malware test lab.

Antivirus Detection Rate (Dec 11, 2006):

‧Fortinet FortiClient Host Security 3.0: 94.33%
‧GRISOFT AVG Internet Security 7.5: 92.75%
‧Dr.Web Antivirus 4.33: 87.44%
‧AVIRA AntiVir ProfessionalEdition 7.0: 87.01%
‧Kaspersky Internet Security 6.0: 83.45%
‧Rising Antivirus 2006: 82.40%
‧ALWIL avast Professional 4.7: 79.16%
‧MicroWorld eScan Internet Security 8.0: 76.31%
‧McAfee Internet Security 2007: 71.55%
‧Symantec Internet Security 2007: 69.96%
‧F-Secure Internet Security 2007: 57.89%
‧CA Internet Security 2007: 55.74%
‧AhnLab V3 Internet Security 2007: 53.78%
‧Trend Micro Internet Security 2007: 53.71%
‧ESET NOD32 2.5: 49.51%
‧Kingsoft Internet Security 2007: 47.21%

[Netherlands]

First off all give us the source.

I don't trust this test. AVG above kaspersky :o F-secure that is using same signatures as kaspersky scores 25,56% lower, Eset almost at the bottum :blink:

There must be something wrong here ???

Ok found it:

http://www.malware-test.com/antivirus.html

[trjam]

http://www.malware-test.com/antivirus.html

I by no means was saying it carried any validity. Lord knows, just looking at it I felt, might add some humor to your day.:D

[WSFuser]

not only AVG is doing good, but Dr. Web too.

NOD32 did poor but at least KAV made the top five.

[EsoxLucius]

It's very interesting how some software didn't manage to enter the tests:
-{ Quote: "Note:
1. When performing the testing, the following antivirus softwares cause some problems so that the testing cannot be continued. For detailed information, please refer to the following attachment.
‧BitDefender Internet Security 2007
‧Frisk F-Prot Antivirus 3.16f
‧Panda Internet Security 2007
‧ZoneAlarm Security Suite 6.5
‧Symantec Norton Internet Security 2007 (hign memory usage) - Source: malware-test.com forums" }-
The reason that some software haven't manage to complete this test seem a little exaggerated. But good job for the av softwares on the top 5.

Ohhh... and BitDefender IS hasn't reached the 2007th version, it's only at the 10th :P

[Pedro]

Seriously, NOD32 is either 95,6% or 45%, DrWeb 58-87, AVG the same, Avast!....

:D

[rdsu]

Do you really think that these tests are trustworthy?

[Bubba]

As it is with their Antispyware Comparison Report....it helps to read their pdf report to understand or grasp their method used. Asking my AV to be robust at adware or spyware is not an item of concern to me.

http://www.malware-test.com/attachment/sha1_for_antivirus_testing_200612.zip.

-{ Quote: "Detailed Test Report
Antivirus Detection Test
Total malware files is 108,236, it includes virus, trojan, worm, backdoor, spyware, adware, dialer, keylogger, hack tool and so on." }-

[Firefighter]

-{ Quote: "http://www.malware-test.com/antivirus.html

I by no means was saying it carried any validity. Lord knows, just looking at it I felt, might add some humor to your day.:D" }-Despite of the detection rankings, I'll believe that those detection rates are far closer that I have met, when some nasties occured to my sight. :P Maybe I just met those odd nasty ones! ::)

Best regards,
Firefighter!

[IBK]

note that the AV's were probably tested using the trial versions, so some AV may had some features disabled (like scanning in archives) while other not (=different settings).

[C.S.J]

interesting test :)

yayyy for my avg and dr.web *hehe* :o

[Londonbeat]

Bizarre results.... :lurking:

[Firefighter]

-{ Quote: "interesting test :)

yayyy for my avg and dr.web *hehe* :o" }-AVG Anti-Spyware module (former Ewido) in the suite is quite good in detecting other than common av stuff nasties, maybe so AVG suite scored so well. :-\

Best regards,
Firefighter!

[FRug]

Looking at the sparse info about the test I want to point out a few things:

1) The tested files come from a honeynet system. This means that unless they have done very thorough checks on the collected malware, it is likely to contain thosands of corrupted samples. This will cause issues for AVs that massively use generic detection, checksum based detection, unpacking and so on. On the other hand, some AVs may benefit from their sloppy approach (fortinet is pretty likely to flag any corrupted UPX file)

2) The testset does not contain files with their original filename, instead they have been renamed by the honeynet to extensionless "honeynet_number" file names. This will cause issues with many AVs if you are scanning with their default settings, which usually are extension based.

3) There is not mentioning of configuration settings, it is however likely that they have used default settings for all applications, which would explain some of the bad results. This would mean extreme detection losses especially for NOD32(no advanced heuristics iirc) and Avira(deactivated heuristics, deactivated modified/strange crypter detection/deactivated SPR category). Also it's not mentioned whether KAV has been using extended or standard databases.

4) The various virus databases have different dates (see screenshots)

5) They have used free/trial versions which may not include support for certain detection categories (i.E. ADSPY and AntiVir Classic)


Now you may decide yourself whether there is any value in this test. IMHO there is none.

[the Tester]

-{ Quote: "Asking my AV to be robust at adware or spyware is not an item of concern to me." }-

I agree.
What's in the testbed would explain the results.

[C.S.J]

deleted

[FRug]

C.S.J:
You are wrong, because his does not reflect their detection rate in real situations. Many AVs by default use extension based scanning, because files without executable extensions cannot be executed, and as such pose no harm to the user. A testbed without extensions is generally frowned upon, ask IBK, Stefan Kurtzhals or The Inspector if you want to. All respected AV tests including IBK, AV-Test.org, Virus Bulletin, West Coast Labs, ICSA Labs use original file extensions for that very reason.

And corrupted malware is corrupted, it cannot be run, and depending on the sample and the AVs detections method it CANNOT be detected (or will be detected by sheer luck depending on where they made a signature). Imagine for example a file with only a remaining UPX packed pe header, fortinet will flag it, becaus it's obviously not a normal UPX file. Most of the other AVs won't make a sound, because simply put, there is nothing malicious in a completely corrupted file. Corrupted samples are the death of a good testbed, ask IBK how much effort he has put in weeding out corrupted samples, i am certain he will agree. An AV is there to detect threats to a users system, not to detect his aborted browser downloads because the file has only arrived 30%...

[C.S.J]

deleted

[IBK]

they may not do that, but they can not do anything against detecting malware even if it is corrupted.

[FRug]

Not intentionally, noone does that intentionally.
Well to drive my point home even for you:
Malware evil_malware.exe is 2MB in size
AV A creates a signature at offset 990KB
AV B creates signature at offset 1100KB
AV C doesn't add a signature at all
AV D adds a signature for the unpacked UPX because they can unpack very well, and to detect possible future variants.

Honeynet downloads 1MB of the file, then the connection/download is interrupted/aborted.

AV A detects the file although it is corrupted and not a threat to the user.
AV B doesn't detect the file anymore
AV C notices there is a UPX header in the file, and the pe header is not matching the file physique -> flag as generic UPX modified evil malware
AV D can't unpack the file because it is corrupted

Neither AV A not AV B has added a corrupted sample, both have added a working sample, by sheer luck AV A still detects the file while AV B doesn't.
AV C will not detect the real threat, but detects a corrupted, non-threatening sample. AV D will also not detect the corrupted file, but will protect against the real threat.

Got the idea now?

[C.S.J]

deleted

[FRug]

No, the test sucks because of its testing methods, not because of the results.

Just look at F-Secure results vs. KAV, they use the same engine, but not the same default settings. You have virtually the same AV here in both cases, yet very different results.

You might want to ask someone at Dr.Web if you don't believe me, but they will certainly tell you the same issues as I did.

[Don Pelotas]

-{ Quote: "interesting test :)
" }-
Yes, if entertainment is the critiria, if an objective result that reflect a realworld scenario is what you are looking for, then .....................::)
-{ Quote: "just because some others are near the top, it MUST BE INCORRECT.

bleah, i dont care." }-
Yes, you do, you're just annoyed that someone like FRug, who unlike you actually have some knowledge about this is disputing the methods in this silly test.
-{ Quote: "
all tests are an interesting read, but you cant put too much into ANY OF THEM, as none are perfect." }-
Thats true, but comparing the AV-Comparatives to this, is like saying Rosie O'Donnell is as hot as Angelina Jolie....................................somehow it just doesn't compute......;D

[C.S.J]

deleted

[Londonbeat]

-{ Quote: "
Thats true, but comparing the AV-Comparatives to this, is like saying Rosie O'Donnell is as hot as Angelina Jolie....................................somehow it just doesn't compute......;D" }-

:thumb: ;D

[Paul Wilders]

-{ Quote: "i was expecting a reply from you don,

kaspersky this, and kaspersky that, its all you hear... boring." }-

Don didn't refer to Kaspersky in his reply in any way - he quite nicely summed up some very valid arguments. So please skip referring to something that's not the case here.

-{ Quote: "well lets say, dr.web, avg, sophos and all them complain about IBK's testing methods, none are perfect and if its malware 'your antivirus' did not detect it, whether how it is or not." }-

That's beside the point here. That said: IBK in fact informs AV companies having their software (freely!) being tested up front as for test results are concerned. As a result (most of the times poor test results) those companies are free to withdraw their software from further testing. As it is very few actually pull out - meaning they do trust they way IBK performs testing. Now, in my book, that's professional testing - contrary to the one being discussed over on this thread, for reasons summed up rather well by amongst others Don Pelotas.

-{ Quote: "another reason, why i ask paul.w to split this forum up into vendors, S&T of all the kasperskys, 'we are better than you', sure." }-

No offense intended, but this has nothing to do with the topic at hand. This topic is about solid and reliable testing - coming with an example how tests should not be performed. At the very least test bed and testing conditions should be out in the open for all to see.

regards,

paul

[marcromero]

Finally, a test with credibility ;D ;D ;D just kidding, of course.

[C.S.J]

deleted

[lodore]

-{ Quote: "Don didn't refer to Kaspersky in his reply in any way - he quite nicely summed up some very valid arguments. So please skip referring to something that's not the case here.



That's beside the point here. That said: IBK in fact informs AV companies having their software (freely!) being tested up front as for test results are concerned. As a result (most of the times poor test results) those companies are free to withdraw their software from further testing. As it is very few actually pull out - meaning they do trust they way IBK performs testing. Now, in my book, that's professional testing - contrary to the one being discussed over on this thread, for reasons summed up rather well by amongst others Don Pelotas.



No offense intended, but this has nothing to do with the topic at hand. This topic is about solid and reliable testing - coming with an example how tests should not be performed. At the very least test bed and testing conditions should be out in the open for all to see.

regards,

paul" }-

wow this is one of the first times i have seen Paul post here for a while. I guess retirement must be fun.
nice post Paul
CSJ can have the lodore syndrome sometimes;D
lodore

[C.S.J]

deleted

[Don Pelotas]

CSJ

Since i obviously hit a nerve..only one more question.

If this is a valid test...then why do Nod32 come out with a 49.51% score?, Surely you do not for a moment think that it would score this low do you?

[trjam]

Just to end this, CSJ, Don is a good man that I have a lot of respect for. I may not always agree with Kas, but I do respect each of these folks for the work they do. Now this will get deleted, but hopefully you read it first. Rein it in man.

[Paul Wilders]

-{ Quote: "wow this is one of the first times i have seen Paul post here for a while. I guess retirement must be fun." }-

Grin...I may not be around that visible over here as used to be, be assured I'm far from retired ;)

-{ Quote: "nice post Paul" }-

At seldom times, I do feel the urge ;D

-{ Quote: "CSJ can have the lodore syndrome sometimes;D
lodore" }-

Never ever heard of such a syndrome ???

regards,

paul

[BlueZannetti]

-{ Quote: "but i ask you this, would those ppl in this thread have come up with these theorys if avira was #1, kaspersky #2 ?" }-C.S.J.,

Looking strictly at the test results only and nothing else, consistency with the established knowledge base realistically is a factor people weigh in assessing information. We do that all the time, including cases where we don't even realize it.

I know that whenever I am confronted with a result, be it an AV test or some type of physical measurement, I do ask whether it is consistent with what I "know". If it's not, then I try to determine whether the result is in error, or whether what I "know" is either in error or incomplete. The problem with many of these published test results is that pursuit of the latter is often next to impossible.

Blue

[oscarlee]

should i swap my nod32 for avg guys or would you keep nod cheers . cause this forum in confuseing
??? ??? ???

[lodore]

-{ Quote: "Grin...I may not be around that visible over here as used to be, be assured I'm far from retired ;)



At seldom times, I do feel the urge ;D



Never ever heard of such a syndrome ???

regards,

paul" }-
Sorry i thought you where retired.

the lodore syndrome is a wilders board joke that trjam started.
I seem to type things without thinking sometimes so once trjam said the lodore syndrome when i did that. so now I make fun of myself really.
but its easier that way. in the f-Prot thread i also used that joke making the point that when companies like norton get sold to the evil corporation like syamantec the support gets bad and doesnt listen to the customers as much.
at that point i switch to the little guy.
lodore

[Londonbeat]

-{ Quote: "should i swap my nod32 for avg guys or would you keep nod cheers . cause this forum in confuseing
??? ??? ???" }-

No, I would not swap anything based on a single test. Any single test.

Londonbeat

[Paul Wilders]

-{ Quote: "Just to end this, CSJ, Don is a good man that I have a lot of respect for. I may not always agree with Kas, but I do respect each of these folks for the work they do." }-

Now, this is a vital remark - not only in regard to Don Pelotas, but in general. Although we do host the Official ESET/NOD32 Support Forums, we do respect other sortalike software companies, their official representatives as well as the volunteers spending time in helping companies as well as their software users out. That has been and will be our general stand, until proven wrong (and that does happen as well on rare ocassions unfortunately). Mutual respect instead of endless controversies ("mine is better then yours" etc.) is what it boils down to. Everyone is free in picking the software(s) s/he likes most and we do expect respect for that.

-{ Quote: "Now this will get deleted, but hopefully you read it first." }-

Although off topic - as is this reply - it won't be deleted; there's no harm having this spoken out.

That said: let's keep on topic from now on - my person included ;)

regards,

paul

[trjam]

Accurate or not, and folks, we went this road with his last results awhile back, if anyone wants to look back on antispyware test results, we need tests, all kinds. Because in the end, it is us, who will decipher the results. And that is what is needed for new members coming here. Facts and solid advice, nothing else.

[tobacco]

You know, i wish he would of tested my 'WinAntivirus Pro'. With those scores, i know it would of finished near the top!. It took awhile, but i'm actually enjoying all these popups. And talk about detection rates, WinAntivirus Pro finds things all the other AV's missed!.;D

[MaB69]

-{ Quote: "
I know that whenever I am confronted with a result, be it an AV test or some type of physical measurement, I do ask whether it is consistent with what I "know". If it's not, then I try to determine whether the result is in error, or whether what I "know" is either in error or incomplete. The problem with many of these published test results is that pursuit of the latter is often next to impossible.

Blue" }-

Thanks Blue for these words of wise

MaB

[Don Pelotas]

-{ Quote: "should i swap my nod32 for avg guys or would you keep nod cheers . cause this forum in confuseing
??? ??? ???" }-
You should keep what you have, a test like this (or any one test) should not make you change your mind.:)

[JerryM]

Keep NOD.
Be very careful about fixing it till it breaks.;D
Best,
Jerry

[EsoxLucius]

Has anyone eventually found out how trustworthy are these tests? What is your oppinion on the test methodolgy?
I found these tests very interesting an somewhat trustworthy, but as I've said earlier the guys that did those tests excluded other software too easy on various weird motives.

[Inspector Clouseau]

-{ Quote: "Has anyone eventually found out how trustworthy are these tests? What is your oppinion on the test methodolgy?
" }-

The reason WHY this test is ******** was already posted here: http://www.wilderssecurity.com/showpost.php?p=900610&postcount=20

AND I SECOND THAT. It is ABSOLUTELY no big suprise that Honeypot samples tend to be corrupted / stripped off! Everyone who has a honeypot knows this!

[pykko]

thanks for telling us the explanation FRug. Not worthy to look at this test.

[Perman]

Hi, folks: Personally I never pay a handful attention to this kind of test. Test results are good only you believe in it. I would treat it as a tea time material, you talk about it and forget about it completely afterwards. Often I ask myself this question; Of virus and anti-virus, which exists first? same question as this; which one exists first, eggs or chicken? And I often wonder that perhaps virus script writer and antivirus app developer share the very same office or are lifetime members of very same fitness club.::)

[JerryM]

Not being very smart in this area (or any other for that matter) I do pay attention to tests. I have decided which I trust. AV Comparatives is the tops for example.
While I do not ignore others it is clear to me that if there are significant differences between the few for which I have high regards, then I try to learn what it is about. If I cannot determine that the tests are for spyware instead of viruses or some obvious reason for the differences I just ignore that test if it is greatly out of line with the good testers.

I really an not interested in how my AV handles tracking cookies and spyware. I have oher applications for that. I do not get too excited if my AV is not always at the top. It is good enough for me, and I had rather use a "known quantity" than play around for another percentage point.

I admit that I think KAV and F-Secure are not going to be far down, and the same can be said for NOD, Avira, and several others.

Best,
Jerry

[Rico]

Hi Guys,

I believe the antispyware testing site was previously referred to by 'Wai Wai', I was rather disappointed by results given to some of my apps. I asked about SAS's poor showing at there forum, & I think I touched a nerve. At least the tone of the replies were first technical (I did not understand the reply) & then defensive in nature, IMHO. Hay! I like SAS & it's currently guarding my system, I also purchased a lifetime contract, I also bought SuperAd.

This antispyware testing site looks very impressive, to a non professional like me. So a few things I do not understand are:

1. The folks at that site seem pretty bright, so what motive would they have for skewing the results? Are they getting paid to have certain apps. at the top of the list?

2. This thread mentions 'honey-pot' a few times, (this will show my knowledge) where/what in there testing methodology alludes, or tips one off that there are indeed using a honey pot?

3. Is there an alternative 'fair' testing site for antispyware. Currently the only one I know is 'spyware warrior.c o m'

4. Has anyone invited 'Antivirus Malware-Test Labs' to lookin on this thread (make them aware of harsh criticism) if they are indeed trying to be legitimate testing sight perhaps some of the thoughts here could be useful for them, to improve.

I found 'AV Comparatives' very helpful in making an informed decision for purchase of my current AV, & this is what I'd like to see for AS's.


Quite awhile back I bought 'Pest Patrol' as somebody claimed this is the best, I ended up dumping it too many 'false positives' for me. Next

I purchased 'Spysweeper' I liked this one, till version 5. Next

I purchased 'AvgAS' I liked this one, but became suspicious, as it never found anything (perhaps this more a testament to my education at Wilders) on my computer. Also some PC magazine trashed AVG/AS. Next

I won a copy of SAS, at 'Calander of Updates' & it's currently at work, along with SuperAD. The purchased SAS given as an Xmas gift

Now I use AvgAS on demand & SAS real time. SAS uses alot more memory than AvgAS. So seeing the results of 'Antivirus Malware-Test' I was tempted to put AvgAS back to 'real-time' & SAS on demand. Thinking I would save a bunch of memory & have better protection. Next

This thread!

Confused in Azusa??? ???

Take Care
Rico

[Macstorm]

Nothing but a smile after reading this report.. :)

--------------------------------------------------------------------------

[MalwareDie]

There are no credible anti-spyware tests in existence as of now. I wonder if there will ever be. This test is no exception I dont believe that it is reliable at all.

[sir_carew]

Strange, I read the latest test and many AV scored different. Bitdefender as the first place, NOD32 second and KAV as third. F-Secure scored different than KAV again, strange.

Link: http://malware-test.com/smf/index.php?topic=3245.0

‧BitDefender Internet Security v10: 94.85%
‧ESET NOD32 2.7: 89.12%
‧Kaspersky Internet Security 6.0: 88.86%
‧Grisoft AVG Anti-Spyware 7.5 (ewido anti-spyware 4.0): 88.50%
‧Alwil avast! 4.7 Home Edition (free): 84.47%
‧Norman Virus Control 5.82: 79.41%
‧Rising Antivirus 2006: 69.11%
‧F-Secure Internet Security 2007: 68.20%
‧Sunbelt CounterSpy 1.5: 67.51%
‧Panda Internet Security 2007: 65.79%
‧Spyware Terminator 1.5 (with WinClamAV 1.1): 65.57%
‧AVIRA AntiVir PersonalEdition Classic 7.0 (free): 57.57%
‧Trend Micro PC-cillin Internet Security 2007: 49.18%
‧EMSISoftware a-square free 2.1 (free): 40.61%
‧McAfee Internet Security 2007: 38.35%
‧Microsoft Windows Defender 1.1 (free): 33.88%
‧ZoneAlarm Security Suite 6.5: 32.12%
‧PC Tools Spyware Doctor 4.0: 31.70%
‧Trend Micro Anti-Spyware 3.5: 28.30%
‧SUPERAntiSpyware 3.4: 23.75%
‧AhnLab SpyZero 2007: 23.45%
‧Comodo AntiVirus 1.1 (free): 17.57%
‧Webroot Spy Sweeper 5.2: 16.51%
‧Symantec Norton Internet Security 2007: 13.33%
‧Lavasoft Ad-Aware SE Personal (free): 12.42%
‧CA Internet Security 2007 (antispyware part only): 9.92%
‧CyberDefender AntiSpyware 2006: 3.08%
‧Outpost Firewall Pro 4.0: 0.93%
‧Tenebril SpyCatcher Express 4.0 (free): 0.52%
‧TrojanHunter 4.6: 0.00%

[Stefan Kurtzhals]

Yeah, another silly test.

He performed an anti-spyware test and tested Avira Classic which does not detect ad/spyware? Yet of the other products, he used the suites or full versions.

[Londonbeat]

-{ Quote: "Yeah, another silly test.

He performed an anti-spyware test and tested Avira Classic which does not detect ad/spyware? Yet of the other products, he used the suites or full versions." }-

I'm impressed with Antivir classic's detection here - it's not even supposed to detect ad/spyware but at 57.7% it hasn't done too bad at all.

Londonbeat

[MalwareDie]

Ya Avira's detection was quite impressive for something that doesn't detect spyware.

[EASTER.2010]

Well, at least it strokes my ego of KASPERSKY IS6 since this is the first AV in sometime that hasn't give my units some sort of issue beit minor or worse.

With all do respect, personally AVG, BitDefender, and Avira have had my attention & my personal interests but didn't qualify to my satisfactions. Hard as it was to regain my complete trust in any AV now that HIPS are on the scene i can confidently say (at least for now) that KIS6 is won me over. I like surprises that lift the spirits compared to lifting anxieties. Taking absolutely nothing away from all the other main players in this field, KIS6 just happens to blend together with my setup while performing as expected and that is reason for some excitement from my camp after experiencing all sorts of frustrations before.

It's a science that walks a very thin line in keeping itself stable while having to live up to what it's designed to do and thats no easy task from what i've seen over the years.

Good effort from all of them.

[Firecat]

I just read this at the Asia Pacific BitDefender site:

http://www.bitdefender.com.sg/NW337-sg--BitDefender-Achieves-Highest-Spyware-Detection-Rate-in-Malware-Test-Lab-Comparison-Report.html

Clearly, if BitDefender made a press release on the fact that they topped this test, then I would expect that BitDefender thinks malware-test.com is reliable. Now this is getting confusing to me....:-\

Interestingly enough, this press release is not there on the global BitDefender page. But regardless, I wouldn't expect BD to make a press release out of something that is supposed to be unreliable (I call that shooting yourself in the foot)....

Of course, there is also the matter that BitDefender apparently did not work properly on the tester's PC for the AntiVirus test, due to which it was excluded.....hmm? Definitely something strange going on here.

[C.S.J]

i know loads dissagree, but whether its corrupted or not etc, it comes to "its either malware or not" and if "one vendor finds malware in its current form, while another AV does not"

i understand the av one was not all put to high settings, but they said they would fix that in the next test, i like the test, its always good to get more people testing, whether some agree or not, they are always interesting to read.

also checkvir.com do a little testing too.