c't magazine 1/2005 AV test

Inside archives I don't even need to check scanlogs with some av:s. Just scan them twice, if after the first scan there have left some infected archives, I can see those not moved/deleted ones. Those that have left, are clean according this certain av. Have you ever tried to count those detected infections from about 3.5k of samples scanlogs, very funny indeed.

Secondly, I just don't want that there is any potential streath in my PC, that's why archived samples.

Best regards,
Firefighter!

 

Don't bother, today NOD32 2.12.3 upd 1.967 with AH scored 218 = 82.9 % against my 263 infected archived TrojanDownloaders. Maybe it's usefull to detect most of the samples from VXHeavens too in certain categories, but I'm pretty sure that I don't even have those all 242 samples from VXHeavens mentioned before in my collection!

Best regards,
Firefighter!

 

Hi,

Please allow me for a moment an off-topic posting

Firefighter,
Is your son safe and OK ?
I can fully understand that your thoughts were at him !
I really hope everything is OK with him !!!!!

Warm regards, Jan.

 

Yes, he is OK, but it was only a lucky night for ME. That storm killed over 10 people here in the Scandinavia.

Best regards,
Firefighter!

 

Hi Firefighter!

Glad to hear both you and your son are ok.

 

My current primary AV and AT are: NOD32 (on-access) and TDS-3 (on-access).

My backup scanners are: Bidefender 7.2 free (on-demand), Ewido free (on-demand), A2 free (on-demand) and AntiVir PE (on-demand) + various anti-adware tools like Spybot S&D, Ad-Aware, Giant, Yahoo Antispy(all on-demand scan only).

The aforementioned do not conflict with each other (on my machine, Win XP Pro sp2), when I set most of them to be on-demand only. Only NOD32/TDS-3 are on-access.

Some anti-virus software do conflict with each other, especially if you set several of them to be on-access. SO they end up fighting about who get's to access the files first/locking them and trying to block each other (assuming malware is at play).

My setup is not perfect and I don't recommend it to others, but it works for me (for now).

For me, it's a decent combination of for-pay and free software that gives me (Imho) a very good protection, without totally bogging down my system resources/speed and giving me a decent amount of user control.

Of course, I don't use all of them all the time, because I try to also practise safe hex and not wildly downloading everything everywhere on the net.

But of course, as always with security efforts, nothing is ever guaranteed

regards,
halcyon

 

Thanks.
I had loaded AVG free on my current machine and it has NAV. The two did OK, but after a couple of days something happened so that I could not get my email. Still not sure what, but when I uninstalled AVG it was OK.

On my new machine I will use Bit Defenrder for my AV, and LnS as my firewall. I will also use A2 free. I may use the paid version if it seems to offer any great advantage.

Thanks again for the help.

Jerry

 

According to the link below,

http://kbase.gfi.com/showarticle.asp?id=KBID002064

Kaspersky can detect more than 900 different packers including variants (= runtime packed files!??).

Let's suppose, that there is totally about 1500 packers including variants waiting for us in real world. According to that we have to test some 230 packer samples including variants if we have set reliability/confidence level to 90 % and Precicion/
Accuracy level to 5 when we want to get a good estimate % to that, how good certain av:s are against runtime packed files.

What does that number of 13 Win32 packers in c't 1-2005 test tell us?

http://scheinsicherheit.sc.funpic.de/avreviews.htm

In my mind it's only more like a result of lottery to test only 13 Win32 packers.

Best regards,
Firefighter!

 

Hi Firefighter,

Is this comparison table you posted available in text format? Possibly on a different webpage? Also did you scan the same files with Kaspersky? Could you please include those results in the table too?

thanks a lot,
-hojtsy-

 

Unfortunately the results are in a "GIF" file. Besides they are a bit too old now. With Kaspersky it's only a bit better than eScan Free but much more work to count detectings. Here you have the recent combo test results. Only Command AV 4.92.8 & BitDefender 7.2 Free combo couldn't beat McAfee VSE 8.0i.

Best regards,
Firefighter!

 

FF-

I am having some trouble drawing conclusions from your tests because there are so many cominations and permutations of 2 AV's or an AV and Ewido possible.

But I can take a shot:

Ewido does almost nothing when combined with a broad spectrum AV such as KAV or its clone, eScan. It helps a little with McAfee, but only in the first category in your test. Beyoind that I can not ascertain the effect of Ewido as BDF, Dr. Webb and Command were not tested on their own in this case.

The next conclusion seems to be that Dr. Webb received the greatest performance boost when combined with Ewido as opposed to Command or BDF, but the difference is not significant. Unfortunately, there is no baseline for Dr. Webb alone on this chart.

The same conclusion as the preceeding paragraph could be said for Command when combined with Dr. Webb, Ewido or BDF.

Likewise, when BDF is matched with Dr. Webb, Command or Ewido, although Ewido causes a more noticable improvement here than in the other combinations.

I would like to be able to say that it makes more sense to add Ewido as a second scanner because it is strong in the first category of your test while some of the AV's are weak, but the lack of single product baselines makes that impossible.

Have fun in Eastern Europe, whoops, Scandinavia.

-Diver (Ron)

 

Here you are, now the test table is corrected!

Best regards,
Firefighter!

 

One small question.
correct me if I'm wrong... how come KAV is a good AV if all one had to do was use an editor to change the signature of a virus/trojan/dropper and release it online.

What I want to know is...
a. Why don't we see a flurry of PCs infected by (for the lack of a better word) non-subtle, brazen attacks on their PCs by only slightly modified malware?
b. What is the motive behind a virus/trojan being "modded"?

I believe I can answer (b) somewhat... The virus modifying hacker (or h4x0r, if you please) will probably be...
1. having a direct, vested, commercial interest in the target computer system/network
2. having an emotional interest... probably revenge.
Both ways... a strong STRONG compulsion to invest TIME, MONEY and EFFORT into such an endeavour.
If the rewards are lesser than the input [or the risks are magnified], people tend to shy away from decisive actions.
Note that the rewards may only be imaginary, as in the case of lotteries.

I hope I haven't strayed OT.. but I thought this was an interesting avenue to explore...

Have fun gang

 

First quote. It's very hard to improve detecting rates with av:s close to 100 % scorings. With McAfee, Ewido of course improved the detection rate mostly among trojan like malware, because it was the weakest point with McAfee and Ewido is an Anti-Trojan.

Second quote. Surprise, it wasn't DrWeb that got the best improvement with Ewido but BitDefender 7.2 Free was the one that got the price. In my mind, a total improvement in detecting rate as high as 10 %, is VERY high. Even NOD32, when it launched the 2.0 version with AH over a half year ago, couldn't manage in this, because the signatures were about the same (against my testbed, NOD32 with best Advanced Heuristics scored some 42 detectings more than with Deep Heuristics only, which is only 1.2 % improvement in detection).

Best regards,
Firefighter!

 

"a. Why don't we see a flurry of PCs infected by (for the lack of a better word) non-subtle, brazen attacks on their PCs by only slightly modified malware?"

It depends.

Replicating malware (virii, worms): modification does not help because AV/AT companies will quickly receive a modified sample and create a new signature.

Non-replicating malware (trojans): there are no statistics on how many vics are infected with modified malware. However, if you know the scene you will be able to confirm that almost all "hackers" try to modify their malware before distributing it.

 

I agree with FF that 10% is a significant improvement, but the significance depends on what is in that 10%. If it turns out to be mostly extremely rare malware or relatively benign "garbage" then it is not as important. Most testers feel that missing only one ITW virus is serious gap.

However, I continue to be amazed that some AV vendors, most notably Grisoft, are intentionally leaving many damaging trojans out of their bases. This used to be the case with NOD32, but they have improved of late.

IMO, the way your tests should look if the AV vendors were doing their jobs right, would be that no combination of 2 AV's or an AV and Ewido would show more than a 4% improvement in detection. No one should need multiple scanners. That is why I like KAV so much. The idea of having two resident scanners gives me a fright when it comes to system stability.

I wonder what the false alarm rate is for these AV's and combo's as well as the effect on system performance (not just memory usage which does not always correlate with system slowdowns.)

This thread illustrates some of the difficulty of testing AV products. At that, it tests only one parameter, overall detection rates. I noticed some debate about the validity of the virus/malware collection. I don't have a conclusion so far as FF's collection goes, but it could be meaningful in some cases. The AV comparatives tests while still placing KAV at the top show a much smaller degree of variation among the different AV's tested than FF's tests. Obviously, it is possible to shrink the collection to the point where nearly every AV is 100% as with VB100 certification. The real question is what exactly are we looking for at the margins and how likely is it to happen.

As much as I like KAV, I have had some files that I thought were scanned a while back turn up as positives later. Fortunately, I did not run these jems. They alsoo turned up positive on a majority of Jotti's tests.

So there you have it, goodies from all over.

 

Just added 37 more samples to my testbed.

Best regards,
Firefighter!

 

All data: Copyright (c) 2005 c't magazine. http://www.heise.de/ct

Heuristic detection of unknown viruses (3 months / 6 month old viruses):

Avast 8,4 % / 5,5 %
AVG 6,9 % / 3,8 %
AntiVir 7,5 % / 5,7 %
AntiVirenKit 27,3 % / 16,8 %
BitDefender 23,5 % / 14,4 %
F-Secure 26,5 % / 16,3 %
F-Prot 9,8 % / 6,4 %
Kaspersky 19,7 % / 13,4 %
McAfee 31,3 % / 19,8 %
Nod32 45,4 % / 33,1 %
Norton 17,8 % / 9,5 %
PC-cillin 3,1 % / 1,3 %
ViRobot 2,9 % / 1,1 %

Nod32 scored best, McAfee was the second-best, followed by AntiVirenKit
Quite impressive to see a test from him showing that Nod32 is the best product in this area!

Scan times (on-demand scanner, copy times of files on-access... the more, the worse):

Avast 216 s / 275 s
AVG 170 s / 208 s
AntiVir 173 s / 313 s
AntiVirenKit 847 s / 1194 s
BitDefender 346 s / 470 s
F-Secure 372 s / 518 s
F-Prot 139 s / n/a
Kaspersky 330 s / 413 s
McAfee 280 s / 340 s
Nod32 150 s / 236 s
Norton 516 s / 764 s
PC-cillin 143 s / 267 s
ViRobot 408 s / 633 s

PC-cillin was slightly faster on-demand than Nod32, as well as F-Prot. Anyway, compared with all other products the speed is amazing high!

Number of online updates per week, (average) size per update:

Avast 4 / 20 KB
AVG 4 / 130 KB
AntiVir 9 / 2450 KB
AntiVirenKit 2 / 860 KB
BitDefender 11 / 360 KB
F-Secure 11 / 485 KB
F-Prot 6 / 2000 KB
Kaspersky 150 / 310 KB
McAfee 2 / 560 KB
Nod32 6 / 170 KB
Norton 2 / 90 KB
PC-cillin 4 / 2220 KB
ViRobot 5 / 1290 KB

It's impressive to see how many KB do you need to download in case of some products... Avast, AVG, Nod32 and Norton scored best in this category!!

c't has tested much more aspects (and confirmed a 100% WildList detection score in case of Nod32 as well as a well-working e-mail scanner), but I can't see Marx' tests are biased in order to trash Nod32.

All data: Copyright (c) 2005 c't magazine. http://www.heise.de/ct

 

I start finding the "who is the best" discussions always more confusing.One time there are some saing the test method is flawed.Another time the tester is biased or didn't use the avs in their best settings.Who should an average user beleive at the end?I ll stick to my AVG for now,since i like having light setup and if i m going to buy an AV in the near future,it's gonna be NOD32 exactly because it's light and good.

But just for my own curiocity i decided to start a completely unscientific statistic method ,using the results of Jottis' scanner.So i started an xls file where i ll keep adding "points" next to the AVs that interest me for every detection i see.It will be random ,but i intend to keep this going for months ,so the bigger the observation sample,the lesser the error.Not that i ll ever buy a bloared AV anywhere,but i want to see what's the difference between some AVs.

 

Does anyone knows the results from the other of the 16 AVs.
I am interested in Norton.

Code:

Software ITW OD/OA Backd./Troj. Heur3/Heur6/Modif.
----------------------------------------------------------------------
Avast Home 4.1.418 100%/98,6% 84,6%/83,3% 8,4%/5,5%/54,8%

AntiVir PE 6.28 100%/100% 66,2%/56,2% 7,5%/5,7%/32,30%

AVG Free 7.0.289 100%/100% 91,4%/31,9% 6,9%/3,8%/32,3%

BitDefender Free 7.2 100%/n/a 99,7%/99,1% 23,5%/14,4%/100%

F-Prot DOS 3.15b 100%/n/a 91,1%/87,6% 9,8%/6,4%/80,6%

Kaspersky Personal Pro 100%/100% 99,7%/98,7% 19,7%/13,4%/48,4%

NOD32 2.0 (1.906) 100%/100% 82,4%/70,2% 45,5%/33,1%/96,8%

LEGEND:
ITW OD = On-demand scan In-the-wild viri (recognition)
ITW OA = On-access scan In-the-wild viri (recognition)
Backd. = Backdoors (recognition)
Troj. = Trojans (recognition)
Heur3 = heuristic with 3 month old signatures (recognition)
Heur5 = heuristic with 6 month old signatures (recognition)
Modif. = Virus modifications (recognition)

 

Think of it like this:

Who do you belive:

1) Professional anti-virus researcher who is accepted/published/respected by the anti-virus community. He publishes under his own name, follows known procedures and has been in the business for a decade or more.

OR

2) An unknown kid from a forum who goes by a pseudonym, more than likely does not have any scientific degree, is probably not a professional anti-virus researcher, doesn't get published anywhere and gets no recognition, except in some obscure forums.

If I had to make a choice between the two, I'd always select option 1 as long as nobody has proven (accusations are not proof) that option number 2 is better in some specific case.

Of course, the ideal situation is where both of the above agree. But if they do not, I'm more likely to believe the professional who's work is subjected to professional peer-review.

I trust the ct review myself. Whether somebody likes it or not, is not my problem

Score for Norton Anti-Virus 2005 (11.0) using the previous 'key' was:

100% / 100% / 93,3% / 85,1% / 17,8% / 9,5% / 100%

Not a bad score overall, but clearly shows that heuristics are the weak spot for Norton.

Also, it loses to the free contenders (Like Bitdefender) in Trojans / Backdoors identification.

Just FYI.

regards,
halcyon

 

In an ideal society where nobody has interest and links with companies etc,i would trust the professionist.In a non angelic society that this doesn't happen,i trust my own judgment which tells me that i don't agree with the following conclusion:

"Conclusion: Not recommended: Avast (both versions) and ViRobot as these scanners did not detect all ITW-samples. McAfee and NAV provided updates too late -> not recommended, too. Fast On-Access-Scanners: AVG, NOD32, PC-Cillin. Good overal protection: Bitdefender Prof., F-Secure."

I have never managed to use Avast more than a week because of the GUI that i don't like and i m using AVG resident right now,but there is no way i am prepared to beleive that Avast detection-wise is not recommended and AVG is indirectly recommended.If the tester is a Professor of univ etc ,nice,but this doesn't make him automatically neither infallible nor above suspicion.I see everyday the university Professors first hand and some would face lawsuits if the people who come to them for their professional aid knew what i know being at the other side of the table.Many times for example it has been posted the way of chosing samples for virus bulletin ,which leaves some doubts on wether it is really objective or not.Also the mere definition of In the wild ,is artificial.I can have a on the zoo virus today,if i release it tomorrow,it becomes in the wild.So at the end ,what interest me and my PC,isn't the artificial definitions of the testers,it's whether my antivirus has good overall chance of cathing any malware that enters my pc,be it virus or trojan etc.

It's not certainly your problem and you did well of posting this here.But it's not my problem either if i have my doubts.If science has progressed it is because everything is not taken from granted just because others hav said it was true before.That's the essense of science.Because any test can be setup and the first law of any statistic experiment is that the sample must be random and representative (2 of the things that Virus Bulletin ,of which i see the Prof is also member, has been accused with reasonable arguments till now).That's why i do my own experiment without worrying about how a tester defines the ITW.For me ,if Jotti's scanner has it,it means i can have it too,so it's enough ITW for my taste and it's certainly as random as it can get.Might not be representative,as it might have more trojans than viruses for example,but after all,when the sample becomes big and comes from real life,i m prepared to take it as good.And in my random test till now,AVG is NOT to recommend.
As you say,that's my view,if the others don't like it,it's not my problem Specially since Internet Security/PCs isn't my profession at all and it's only a hobby for me so i can enjoy all tests etc and judging them from my own prespective of what i want and intend as secure product for my PC

 

Also ,given the opportunity,can someone enlighten an amateur hobbyist on this doubt:

Avast Home 4.1.418 100%/98,6% 84,6%/83,3% 8,4%/5,5%/54,8%

AVG Free 7.0.289 100%/100% 91,4%/31,9% 6,9%/3,8%/32,

Why should Avast be not recommended and AVG be,when AVG has that horrible 31,9% on trojans?I mean,is it more probable to catch the 1,4% that Avast won't catch ITW or the 68,1% of AVG?It can be a coincidence but the few times i had an infected file in my pc ,it was a trojan.
So the problem is,how do these stats represent real life?Virtually all appear 100% ITW ,so no user should be infected.
But then why do the same scanners rate so differently in Jottis' scan?Before i was there and AVG didn't detect NetSky.C mail Worm.It's not in the wild i guess.But it was ITW enough for the person that submitted it.

That's why i have my doubts of all these ITW definitions,what are their real value for real time computing,why people still get infected even with updated AV if "ITW" even AVG is 100% etc.
Maybe Jotti has all the weird submissions from in the zoo samples and that's why so many scanners fail.But i start to beleive that this is not the case and instead there is something wrong with the beraucratic procedure that dictates what are the rules of ITW and what not and why Avast with 98,6% ITW and 80+ in trojans is not recommended while there is no negative reccomendation for AVG.I just know that at jotti's scanner,there is a real battle between AVs in every sample and RARELY all of them detect it.And in real life,i just want the AV to catch as much as possible.I don't care if it was in the sample "ITW" or not or if it's trojan or not.I want it caught.

 

If thats the case,KAV is your toy to play You can back it up with NOD32 (i haven0't tested if you can run them both at same time).
This way you get perfect combination. KAV for thorough overall detection and NOD32 for latest threats if KAV somehow fail to update in that timeframe of outbreak.

 

Nah,i had KAV 4,5 and 5.0 and then uninstalled it because of the ADS.I caught a byteverify that KAV saw one instance and AntiVir 4 i think.At the end,i m not infected easily (last time i really got something important was with Blaster because i had stopped AVG resident and the firewall for 20 seconds,then AVG saw the virus but coulnd't clean it so i formatted).It was nice having KAV but once i uninstalled it i could open folders visibly faster.

My post is my questioning in general of all this testing.I mean,something is ITW if it's "spreading from PC to PC with the user unaware and at least 2 experts report it".This somehow doesn't sound very reassuring or gives right to detection of programs.As you know Rejzor,i ve never been a fan of Avast,but i think it's unfair to say that AVG got 100% ITW so it's better AV than Avast detection wise.

Anyway,i ll leave you experts continue your debate on the results and i ll sit back and enjoy the massacre