c't magazine 1/2005 AV test

Firefighter,

Thanks, I should have realized that, but I didn't.

I notice some have problems with Ewido, and so I am reluctent to install it. I do have A 2, but I understand it is not as thorough in malware as Ewido.

I have not hooked my new computer up yet, but have downloaded Ewido. I thought I would give it a try on this computer, but it would not install on W 98.

Thanks, I really do appreciate all the information and help I get here.

Jerry

 

@firefighter

I have downloaded the TrojanDownloader.Win32 section. It contains 225 files. I scanned it with NOD32 AH (signature database dated 15 October 2004) and allowed NOD to delete any detected files. 60 files remained.

Then I updated the signatures and conducted another scan. 33 files remained.

The decisive question is whether a significant percentage of these 33 files can be qualified as malware.

The non-detected files are:

TrojanDownloader.Win32.Apher.a
TrojanDownloader.Win32.Aphex.060
TrojanDownloader.Win32.Dyfuca.ap
TrojanDownloader.Win32.Dyfuca.bt
TrojanDownloader.Win32.Glukonat.a
TrojanDownloader.Win32.Harnig.a
TrojanDownloader.Win32.Herman
TrojanDownloader.Win32.IstBar.dv
TrojanDownloader.Win32.Livup.a
TrojanDownloader.Win32.Miled.a
TrojanDownloader.Win32.Mosw
TrojanDownloader.Win32.QDown.d
TrojanDownloader.Win32.Realtens.g
TrojanDownloader.Win32.Small.ew
TrojanDownloader.Win32.Small.ga
TrojanDownloader.Win32.Small.jn
TrojanDownloader.Win32.Small.q
TrojanDownloader.Win32.Smokedown.a
TrojanDownloader.Win32.Swizzor.t
TrojanDownloader.Win32.Swizzor.u
TrojanDownloader.Win32.VB.aa
TrojanDownloader.Win32.VB.aj
TrojanDownloader.Win32.VB.bb
TrojanDownloader.Win32.VB.cl
TrojanDownloader.Win32.VB.cn
TrojanDownloader.Win32.Webaut.g
TrojanDownloader.Win32.WebDL.02
TrojanDownloader.Win32.WebDL.d
TrojanDownloader.Win32.WebDL.f
TrojanDownloader.Win32.WinShow.aa
TrojanDownloader.Win32.Wintrim.aq
TrojanDownloader.Win32.Wintrim.as
TrojanDownloader.Win32.Wintrim.v

Please note: I know that you have been bashed @ Rokop. But my intention is certainly not to do the same. I have no clue whether these files are indeed malware or not. Therefore, it may very well be the case that it DOES make sense to use (at least certain parts of) the vxheavens archive for testing purposes. Let's continue to figure it out ...

 

O.k. ... I am digging thru the files right now (not performing a detailed analysis).

TrojanDownloader.Win32.Apher.a --- just a harmless client.
TrojanDownloader.Win32.Aphex.060 --- just a harmless client.
TrojanDownloader.Win32.Dyfuca.ap --- seems to be a porn-related adware (maybe a dialer) ... but not a real trojan (at least not a working one).
TrojanDownloader.Win32.Dyfuca.bt --- seems to be a porn-related adware (maybe a dialer) ... but not a real trojan (at least not a working one).
TrojanDownloader.Win32.Glukonat.a -- just a harmless hack tool (hotmail account locker).
TrojanDownloader.Win32.Harnig.a --- seems to be corrupt/damaged/not working.
TrojanDownloader.Win32.Herman --- does not seem to do anything (except establishing an outgoing connection).
TrojanDownloader.Win32.IstBar.dv --- seems to be a porn-related browser hijacker (that adds a toolbar to the internet explorer).
TrojanDownloader.Win32.Livup.a --- unclear...does not work?
TrojanDownloader.Win32.Miled.a --- seems to be corrupted/damaged.
TrojanDownloader.Win32.Mosw --- harmless client/visible (not stealth)
TrojanDownloader.Win32.QDown.d --- does it work??
TrojanDownloader.Win32.Realtens.g -- harmless client/visible (not stealth)
TrojanDownloader.Win32.Small.ew --- does it work??
TrojanDownloader.Win32.Small.ga --- does it work??
TrojanDownloader.Win32.Small.jn -- visible client
TrojanDownloader.Win32.Small.q --- does it work??
TrojanDownloader.Win32.Smokedown.a --- harmless client (visible)
TrojanDownloader.Win32.Swizzor.t -- does it work??
TrojanDownloader.Win32.Swizzor.u --- does it work??
TrojanDownloader.Win32.VB.aa -- Free history cleaner 2.77 (visible) - false alert or a dialer?
TrojanDownloader.Win32.VB.aj ..... too tired to continue.
TrojanDownloader.Win32.VB.bb
TrojanDownloader.Win32.VB.cl
TrojanDownloader.Win32.VB.cn
TrojanDownloader.Win32.Webaut.g
TrojanDownloader.Win32.WebDL.02
TrojanDownloader.Win32.WebDL.d
TrojanDownloader.Win32.WebDL.f
TrojanDownloader.Win32.WinShow.aa
TrojanDownloader.Win32.Wintrim.aq
TrojanDownloader.Win32.Wintrim.as
TrojanDownloader.Win32.Wintrim.v

In the light of the above, I ask myself whether NOD32 should be bashed for not detecting these samples. It seems that most of the samples do not download a real trojan but do something else (or nothing). Taking into account that these samples are categorized as trojan downloaders I continue to argue that the quality of the vxheavens archive is sub par.

Btw.: One of the above trojandownloaders also included PowerCleaner...an (adware/spyware?) application designed to remove porn-related files ;-)

 

Well I wouldn't want any of that rubbish on my pc and an AV prog that didn't (couldn't) even warn me about them. So to me they are valid.

 

@IanB

As I said before ... different users demand different things from a scanner. Many users just want to get rid of any rubbish. But there are also users who do not want a computer program to decide what's "rubbish" or not. They merely want to detect a computer program DANGEROUS malware.

Moreover, I believe that it is of importance to note that a tester using the vxheavens archive cannot reliable determine WHAT kind of malware is (not) detected by scanner. This is bad because a good tester should exactly know what s/he is testing. If a tester does not precisely know what s/he is doing s/he cannot properly analyze and/or describe a scanner. S/he can just provide some meaningless figures.

 

TrojanDownloader.Win32.Dyfuca.ap --- seems to be a porn-related adware (maybe a dialer) ... but not a real trojan (at least not a working one).
TrojanDownloader.Win32.Dyfuca.bt --- seems to be a porn-related adware (maybe a dialer) ... but not a real trojan (at least not a working one).
TrojanDownloader.Win32.Herman --- does not seem to do anything (except establishing an outgoing connection).

They seem dangerous enough to me.

I agree to an extent. Nevertheless, I for one appreciate the time and effort that Firefighter puts into his tests (I've tried it myself and it can be a real effort). He doesn't proclaim that his tests are the "be all and end all" but they are INDEPENDENT and a welcome source of information (not all of his collection is VX).

 

"Nevertheless, I for one appreciate the time and effort that Firefighter puts into his tests"

100% agreed! This is NOT about bashing Firefighter. This is merely about improving tests and figuring out things. Nobody is perfect. Therefore, it is important to be open-minded. The worst testers are those who do not disclose their test procedure. Firefighter was brave enough to do so and, consequently, there is the chance to learn.

 

I'm not saying that you NOD folks have quite low ego but why you just can't accept that ONLY eScan Free, Norton 2005 and DrWeb were better av:s against my TrojanDownloader samples? NOD was better than McAfee VSE, Command AV, BitDefender, Avast 4.5, AntiVir, AVG etc.

How often we have to see complaints like these? NOD still scored almost 78 % against TrojanDownloaders of mine. From those samples you mentioned just now only ONE was detected by Kaspersky only in Jotti's online scan. Please, submit those "Clean" files to VirusBulletin, ICSA Labs and Checkmark (= West Coast Labs), that they at last can do a proper false positive test against clean files so that there isn't so many av:s to get these certificates.

Best regards,
Firefighter!

 

It's not about NOD. NOD maybe the crappiest scanner in the world (or not). I don't care. I picked NOD because the detection rate was rather low and it was readily available (i.e., it was already installed on one of my test machines).

We shouldn't get distracted: my comments are about testing methods and the samples from vxheavens. Not about NOD.

In my opinion, the decisive question is not whether these files are "clean" or not. By contrast, the question is whether the files are indeed trojan downloaders (or harmless clients, semi-dangerous spyware, corrupted malware or something else which may be dangerous or not). In other words, the question is whether a tester should know what s/he does ... in particular whether a tester should know his/her own test archive.

 

Here you are right. There isn't such thing as be right to make categories. When I looked at my logs, each av-vendor is categorizing these samples differently. But who has the right categories. Most of my samples are categorized by Kaspersky of course, because almost every available sample collection is verified by Kaspersky.

About NOD's detecting rate, among trojanlike malware as a whole, only those av:s that I mentioned in my last post plus McAfee were the only that were able to detect more, so in my mind that's not poor result, when NOD's result was dated about two weeks earlier than the latest test results dated 29. Dec 2004.

Best regards,
Firefighter!
_____________

 

This means nothing really. Your samples are limited and should not be used for AV judging. I do applaud for your efforts though.

Think it this way. If you or anyone can get your/their hands on these samples without any problems (from well known VXer site) then what’s stopping AV research teams?

NOD32 has ability to detect Viruses or Trojans without need of updating which is something you could actually use to judge AV program strength.

Not just NOD32 but any other AV has its strengths and weaknesses.

When it comes to overall detection (signatures) of viruses, KAV is your best bet. Because of this KAV is bloatware and still misses viruses (see picture), where NOD32 lightware heuristically picks it in this example (see picture). <ahem, this is a typical NOD32 fanboy set up, so only NOD32 could detect this Trojan or something>.

In general good detection rate (note: GOOD not the best or very good) combine with loads of great features and your brain will be you best bet.


tECHNODROME

 

In my opinion, it's almost useless to put 1,000,000 viruses/worms/trojans/malware to a test if a tester doesn't really know how circulation state in the real world of all those malware. Detecting 99% or 100% of all 1,000,000 trojans/malware looks good but it has nothing to do with real-world protection, not all 1,000,000 trojans/malware circulate in the real world and not all of them cause real danger to every group of users.

Detecting malware (not specified to only viruses/worms in the WildList) that are still circulating in the wild and the abilities to detect new dangerous malware as soon as posible or without any updating by not introduce unacceptable slowdown or any glitches are something we should actually use to judge AV program strength, not by illusive best overall detection rates or big database.

 

Some day, when I have enough of time, I'll make a total new kind of REAL ItW test.

Because my PC is already allmost a crap now, what do you think of samples that I can get by reinstalling the whole WinXP Home? After that just disabling the WinXP Firewall, making no updates to WinXP Home (about 4 year old system) and using any other firewall and antivirus.

Surfing in the web about one week with unpatched Explorer without downloading anything, just surfing. During my test I don't shut down my PC. Do you think that I can get some interesting files when I have a cable modem connection in my PC and I'll make some scans after that with eScan Free first, then with Ewido and report only position enabled, then with DrWeb, NOD, MKS_VIR, McAfee VSE and so on?

Best regards,
Firefighter!

 

@TAP: you did hit the nail

 

Moreover, what's the benefit of detecting 3 billion zoo trojan.downloaders if it takes a real-world attacker only three seconds to load malware into a PE editor and change the entry point so that KAV (and any KAV clones) can't detect it anymore:

http://img132.exs.cx/img132/5900/kav7cb.jpg

@firefighter Download Lord PE or another PE editor and try it out yourself. The KAV scan engine is completely flawed and should not be used if you want to reliably detect non-replicating ITW malware.

 

To be fair here this is not a case with KAV only. The same thing will happen to NOD32, NVC and etc.

If you are concerned about this then you should go with DrWeb. Its kind hard to fool it but not impossible...


tECHNODROME

 

@Technodrome

I disagree. NOD32 is not affected provided that AH is enabled. Dedicated trojan scanners like Ewido or BOClean are also immune. (I agree that NVC is affected. But it's crap anyway ;-)

 

I meant without AH. But what about infected files which AH is not able to detect ?


tECHNODROME

 

I'm very sorry if you think that I'm a spokesman to some av-vendor like Kaspersky from which you just picked your example. My purpose was only to show the overall RANDOMLY picked samples detection rate. I use McAfee VSE 8.0i as my resident av and DrWeb 4.32b and CSAV 4.92.7 as my backups. My Anti-Trojans are BOClean 4.11 and Ewido 3.0. So what? These are only security programs among which we just can't find any proggie that is infallible or invulnerable.

If anyone makes tests from sample collections downloaded from web, over 80 % of those samples that we can download are verified as infected by Kaspersky. That's why I tried also pick some new samples from different forums and some samples from those writer sites too.

But overall about Kaspersky. There is some reason why almost all sample collections are verified as infected by Kaspersky. Maybe you already know why?

Best regards,
Firefighter!

 

Well, there are certainly many files that KAV (or other scanners) can detect and NOD32 cannot detect. And vice versa.

The good thing about NOD32 /w AH is that it is DIFFERENT from KAV & clones. Therefore, it nicely compliments KAV and many other scanners.

I would be interested in a test that accurately characterizes scanners so that it is possible to pick a good combination of scanners.

 

@firefighter

No worries. I do not believe that you are a biased tester. (And I am not a NOD freak.) I just wanted to open your eyes. I would be glad if you further improved and continued your tests.

 

Then I need scanners that are able to delete or move infected archives, because my samples are in zipped files.

Btw, I'm not so found of KAV and some other combos, because tests like mine have always a bit skewness to KAV because of that sample picking methodology. But I'm quite sure that this kind of skewness is almost in every "Zoo" test I've seen.

Best regards,
Firefighter!

 

"because my samples are in zipped files."

Why don't you unzip them? The test results might be invalidated by using zipped, rared or aced filed.

This is because it's highly questionable whether a scanner must be able to scan inside archives.

 

I think that all reasonable scanners are able to scan inside Zip files, if not, I don't care. Here you have one av-combos, DrWeb & Command AV compared to eScan Free 4.7.6 and McAfee VSE 8.0i. Not bad combo at all in my mind.

Btw, inside archives you can count more easily those infected archives compared to infected samples because many scanners are reporting at best some 10 infected files in one sample (BitDefender, DrWeb etc.). Very funny to check these scanlogs.

PS. Im sorry about my a bit hostile posts to you today, but my oldest son (19 year old) took my car to pick some girls out of the restaurant 4 AM and the distance is about 20 miles in the middle of nowhere and there is a very big winter storm coming here just now, just worried about my kid. What is the power of women to men in situations like this?

Best regards,
Firefighter!

 

"I think that all reasonable scanners are able to scan inside Zip files, if not, I don't care."

I understand from vlk that this is exactly the attitude of A. Marx: some testers do not care whether a scanner properly performs ITW. They consider it more important whether a scanner accomodates to their personal testing needs. The detection of archived malware is not of utmost importance since the on-access scanner will detect archived malware as soon as it is unpacked (and before it is executed).


"Btw, inside archives you can count more easily those infected archives compared to infected samples because many scanners are reporting at best some 10 infected files in one sample (BitDefender, DrWeb etc.). Very funny to check these scanlogs."

Frankly speaking, I don't understand this comment. What is the advantage of scanning archived malware? Why do you want to determine the ratio between infected archives and infected samples (contained in such archives)? Moreover, I hope that you do not determine a scanner's detection rate by looking at the scanner's scan log. The scan log's are not reliable and frequently contain a number of infections which is too high. (For example, archived or run-time compressed samples might be double-counted.) You need to allow the scanner to delete any infected samples. Then you can compare the number of original samples with the number of remaining samples. You can also create a reliable scan log with the help of a tool like dirlister.