Scheinsicherheit's test against trojans finally completed!

Hi everyone! I have just seen those test results against trojans, where the finally ranking among 5 AT and 112-113 trojans were:

1. (2.) 58 TDS 3.21

2. (1.) 27 (62 as heuristics) TrojanHunter 3.5

3. 26 Trojan Remover 4.9.7

4. 14 Anti-Trojan 5.5

5. 7 PestPatrol 4.0

http://scheinsicherheit.o-f.com/scanners.htm


Best Regards,
Fírefighter!

 

From that page:

which last part ....... !!!
Please don't waste your time there.

 

Jooske, what is your problem with these tests?

Do you think, they are wrong in some way?

 

@Jooske

Thanks for citing our "disclaimer". It is really of critical importance not to get confused by the numbers.

Of course a scanner which detects 110 out of 113 compressed malware samples has probably a better unpacking engine than a scanner which detects only 1 out of 113 compressed samples. But it is definitely not a good idea to say that, for example, Trojan Hunter is "better" than TDS-3 just because its file scanner has detected a few samples more. (If you translate the entire report you will see that we also try to analyze a scanner's signature quality etc.). Moreover, it must be noted that both Trojan Hunter and TDS-3 also support memory scanning (and not only file scanning like other AV/AT scanners).

I would appreciate your criticial comments regarding our site. ("Please don't waste your time there" ... seems a little bit too general to me.)


@Firefighter

Unfortunately, our standard test archive is outdated. That's why we also perform supplementary tests with special samples (like DLL trojans) that are not included in the standard archive. The new test archive will include DLL trojans and several new compressors/crypters. It will also demonstrate that the unpacking engines of several well-known scanners (like KAV) can be easily outfoxed by compressed samples with a modified unpacking stub.

I am unhappy that the most important parts of our website are still not translated (including the reports). However, translating all this stuff is really a pain in the neck ...

Cheers,

Nautilus

EDITED:

I have just read Wayne's post re the "vulnerability papers". He says:"The author(s) of those articles seem intent only on one thing - trying to make a program look as bad as they can."

Because some people may come to the conclusion that we do exactly the same on our site (i.e., running down AV/AT software) I would like to note that we do not generally reveal any vulnerabilities which are not already discussed in the trojan scene etc. For example, the issues with Trojan Hunter were revealed by "Catinahat" on 23 January 2003 (i.e., months before we finalized the TH test mentioned by Firefighter). We discussed these issues (and certain others) because we believe that not only VXers but also the customers of a scanner should be informed about them.

 

As I cannot read the full report it would be interesting to know whether the Trojan scanners tested were set in their default configurations or with everything switched on and at the highest level i.e. TDS3's six startup scans & exection protection? Was the Generic Detection all set to enabled with heuristic scanning sensitivity set to the highest position?

Befor issuing this report, have you issued a list of the Trojans + variants tested to the progamme developers? If so what feed back have you had?

Not knocking the tests just interested in the test conditions and communication

 

@Pilli

1.
We try to get the max out of each scanner which is sometimes not so easy. Did you know that TDS-3 features a (basic) manual module scan which cannot be automatically switched on? I did not ;-)

If we play with the TDS-3 mem scanner we use both object and process memory scan. (Note: The numbers cited by Firefighter only relate to the TDS-3 filescanner. We expressly mentioned that the mem scanners of Trojan Hunter and TDS-3 detect almost every compressed trojan. But not every hexedited trojan.)

I believe that the TDS-3 heuristic is set to the highest position by default. At least it is set to this position on my test machine. Execution protection is not generally switched on. (I made the experience that execution protection will not help if the file scanner can't detect a trojan. Moreover, the trial version does not support execution protection.)

2.
We used to inform the developers before issuing a report. Some of them asked us to translate the report. Some of them ignored us. Some of them wanted to get the uncompressed samples which were missed by their scanners. Others asked for the compressed/crypted samples in order to create new signatures (which does not always make sense because our objective is to determine whether a scanner has an unpacking engine or not). As regards TDS-3, I remember, that Wayne asked for a translation and recommended to read the DiamondCS test guidelines ;-)

In summary, I believe that it does not make much sense to inform the developers before issuing a report. We usually inform them about specific findings. For example, we notified DiamondCS that TDS-3's heuristic detection of a trojan's ICQ notifier was based on a case sensitive string. This was immediately changed. Therefore, we felt no need to mention this issue in our report.

Nautilus

 

Nautilus, Thank you for your curteous reply.
I am sure that all the developers are working very hard on their products unpacking capabilities.

 

I jumped over "a usable unpacking engine"
analysing the software as you did you would have found them; further if you are a registered TDS operator you would have found certain more capabilities in the software of which among others the install of exec protection.
Further in the members only private forum we are educated in adding more unpacking engines if we like besides the several ones (USABLE and working!) included in the software, while waiting for the TDS-4 family.
The worm sensitivity slider can be moved all to the right.

TDS is not an out of the box and press the button product.
First after installing one updates the radius database, if the version is registered it's just a press of the button.
Then one looks at the configuration, the scanoptions, one edits the scan files how we want them done and what to include, do we want the whole network or only the pc on which it is installed, etc.
etc etc etc

For the test i suppose the scanning is set to optimal for the registered version with all possibilities set to maximum.
The kind of system it's running on we want to know, OS, RAM, etc.
What i expect in a technical test (TECHNICAL) is how the definitions database is made and detecting what it is supposed to, how it is done, how it is detecting, how the defenitions database itself is protected or which form it has, etc. Can we read it all as a textfile, is it encrypted, other ways which can be imperative.
And of course in all cases updated to today's last signatures.
Nice that the test changed the ICQ definition, good for all of us.
Such details are interesting if send to the developer when found so they can fix those finds where possible/necessary.

Both english nor german are not my native languages, so for such translations best to have one who is either very good in both or native in one of them.
Sometimes i translate little pieces so we all can understand the general sense.

But never tell again TDS wouldn't have included useable unpackers for this is absolutely untrue.

 

@Jooske

1.
"I jumped over "a usable unpacking engine"
analysing the software as you did you would have found them;"

Let's face the truth. Even Wayne and Gavin do not dispute that TDS-3 does NOT have a sophisticated unpacking engine. We DID properly analyze the software. For example, we used several TheefLE samples which were packed with different UPX versions. We also used different compression settings. TDS-3 decompressed only one out of five UPX compressed TheefLE samples (UPX.theefLe111.upx124_comp8.exe).

BUT: Please note that we expressly mentioned that TDS-3 uses alternative scanning techniques like heuristic scanning (and its mem scanner of course). In a recent thread @ Rokop Security I examined how AV scanners with a decent
unpacking engine handle compressed samples with a modified unpacking stub (see my post @ http://www.rokop-security.de/board/index.php?showtopic=682&st=15 ) The results were quite disappointing. Amazingly, TDS-3 detected more samples than any other scanner. Not because TDS unpacked the samples but because its
scan heuristic was able to identify them as malware.

AFAIK TDS-4 will have basic unpacking support.


2.
"further if you are a registered TDS operator you would have found certain more capabilities in the software of which among others the install of exec protection."

I found no difference when I enabled execution protection. I believe that execution protection does not increase the detection rate but prevents malware from being executed.

3.
"Further in the members only private forum we are educated in adding more unpacking engines if we like besides the several ones (USABLE and working!) included in the software, while waiting for the TDS-4 family."

This is mentioned in our report. We also mentioned that other scanners provide unpacking support right out of the box.

4.
"The worm sensitivity slider can be moved all to the right."

Correct and that's where my slider is ;-)


5.
"For the test i suppose the scanning is set to optimal for the registered version with all possibilities set to maximum."

I do not believe that the registered version has a better detection rate. DiamondCS would probably mention such a difference because a "bad" trial version could damage the reputation of the product.


6.
"The kind of system it's running on we want to know, OS, RAM, etc."

Such information is provided. OS is always WinXP. Different computers were used including VMWare virtual machines. (Every scanner was installed on at least one "real" computer.)

7.
"What i expect in a technical test (TECHNICAL) is how the definitions database is made and detecting what it is supposed to, how it is done, how it is detecting, how the defenitions database itself is protected or which form it
has, etc. Can we read it all as a textfile, is it encrypted, other ways which can be imperative."

Such information is provided. For example, we mentioned and critized that Trojan Hunter's signatures are not protected. We did also examine whether a scanner uses weak signatures (like text strings). Moreover, we explained that Kaspersky's signatures can be extracted with a well-known hacker tool. We did not attempt to decrypt a scanners signature database by using SoftICE. And I think most software developers are happy about this. (In particular,
Wayne was concerned that we would try to do so with TDS. But we did not.)


8.
"And of course in all cases updated to today's last signatures."

The date of the last signature update is included in the report. Sometimes we performed additional tests with outdated signatures because this allows you to determine whether a scanner's heuristic detection works fine.


9.
"Both english nor german are not my native languages, so for such translations best to have one who is either very good in both or native in one of them. Sometimes i translate little pieces so we all can understand the general
sense."

Native speakers are welcome to translate the rest of the page


10.
"But never tell again TDS wouldn't have included useable unpackers for this is absolutely untrue."

Sorry. I still disagree. I can send you several UPX samples which cannot be uncompressed by TDS-3. (They are not modified in any way.)


Cheers,

Nautilus

 

Best send them to support@diamondcs.com.au as they can look in the most technical sense to it and if necessary can change what's possible in the detection databases. Thanks for the submission, for the wellbeing of the internet community as a whole.

Your "unpackers" remark should have some refining "in our test for these samples X, Y, Z"
as there are very valid and working unpackers included.

 

To Jooske from Firefighter!

You said: "TDS is not an out of the box and press the button product". I absolutely agree.

In my mind every product has to prevent all possible danger situations with default settings. When program's settings are too difficult to an average user, is it the user's fault or a mistake during that process when the program has been released?

It is at least a good advice in marketing announcements when people should been told that this product's main purpose is to help IT PROFESSIONALS in their daily work, not for common people!

After that all these kind of products has to be tested among IT PROSSIONAL's tools, not within for excample AntiTrojans.


"The truth is out there, but it hurts"

Best Regards,
Firefighter!

 

Hi Firefighter,

I'm afraid you are simplifying matters too much here.
TDS-3 out of the box takes as many buttons to click to perform a system scan as most of the other AT scanners.
No hocus-pocus or trained IT-pro required.
Granted, it takes a little more time to use the more advanced options (which most AT scanners don't have, no matter how many buttons you push)

Regards,

Pieter

 

Let me put it differently:
first look at this thread:
http://www.wilderssecurity.com/showthread.php?t=12743

After installing and updating you can choose for press the buttons, but as TDS leaves you in the drivers' seat and never in the dark you get every help and instruction to finetune and even do it better for your circumstances and special wishes, while you're top safe in all conditions.


But that part of the discussion can be done in the suitable TDS forum, here we talk about the tests and to unpack or not unpack and other parts.

 

See here for my response a few days ago to this report.

For the record, TDS3 was the first anti-trojan scanner with any unpack support, but in regards to TDS4, I can't go into details but rest assured unpack support is well and truly taken care of, we've spent a lot of time and money ensuring that, like nobody else has.

You'll soon see, and we thank you for your patience - we hope to reward that soon.

 

Just to extend on that ....
Nautilus: > AFAIK TDS-4 will have basic unpacking support
Advanced or complex is the word you're after here, not basic . I wish I could elaborate more on this now but I know you'll understand why I can't, but it won't be long before you can try it for yourself.

And in regards to not being able to use TDS out of the box, why not?
1. Download it, install it.
2. Grab the latest databases to ensure you're up-to-date.
(Those first two steps apply to every anti-trojan scanner, so nothing new so far)
3. Start TDS, click on the System Testing menu, then Full System Scan.
Two mouseclicks. It can't be simplified much further than that.

The common misconception with TDS3 Professional is that because it offers the user so many advanced functions it must be hard to use, but it's only as hard to use as you want it to be - most things (including updating the database, scanning your entire system, etc) only take a couple of mouseclicks. If you want to use the advanced utilities (most of which can't be found in any other anti-trojan packages), then you can, and mastering such extra tools will help you detect unknown trojans and other malware. You don't have to use these extra utilities though, it's up to you, but most anti-trojan packages assume that everyone using it has little experience with security - TDS3 on the other hand also caters for security experts and people who have a keen interest in all facets of the security of their system.

 

To Pieter_Arntz and Jooske from Firefighter!

All that I meant before was that 99.9 % of all tests I have seen were done with more IT skills than I never can adopt.

After all this when we were told that those people couldn't take the full performance of certain programs, for me that's not the testers fault but more and more the product's own feature.

What value does "the voice of the customer" actually include?

"The truth is out there, but it hurts!

Best Regards,
Firefighter!

 

FF, your reaction crossed Wayne's explanations.
I, as a normal user, am not even interested to be very professinal to perform all kinds of advanced tests, i just use the two mouse clicks scanning and in time i might have refined some settings, for the rest when needed i use more techniques included in the Suite and we licensed operators for sure have added some more functionallity with scripts, working on them as a team where possible.
But the click-click scan is still there, even though there are people who worked on having even that process automated.

At times i walk over different online scanner sites and have all the system tested, to find nothing new and if so i sent the samples to find after a report of "false positive" from the techies and making such online scanners happy with those comments. Believe me, my system is used and tested a lot!
This is no ordinary windows OS, better call it a test-case.


BTW FF: for your signature, you might like to add something like " the truth lies out there and it doesn't hurt me as i know myself well informed and protected with top notch software!"

 

Just for the record ;-)

1.
I did not know about the topic in the DSL forum since I was offline for a week.

2.
We did not rate TDS-3 #1 (nor did we say that it is not #1). There is no rating. You have to read the report (i.e., the scan results and! our comments) in order to make up your own mind.

3.
There is a big yellow disclaimer on our site. Unfortunately it is not translated yet. Basically, it says that we did not perform a complete AV/AT review at all !! This is because we mainly checked the scanners' decompression capabilities, signature quality and certain other issues which are not discussed in ordinary AV/AT reviews.

A good AV/AT scanner needs to meet a lot of criteria and we have checked only a few of them. Therefore, you can only draw a reverse conclusion from our report. A scanner without a sophisticated unpacking engine (or alternative scanning techniques like Trojan Hunter or TDS-3) can't be recommended as a trojan scanner. It may still be a good scanner for replicating malware.

If you want to know whether a scanner has a comprehensive signature database you should have a look at ordinary AV/AT reviews. We do not perform such tests (although we mentioned that TDS-3 uses one of the most comprehensive and up-to-date signature databases).

4.
@Wayne

"Also, they're comparing heuristic detection with positive IDs, but this can't happen because when a positive ID is made, heuristic scanning isn't required - it's redundant at that point. For example, 58 trojans were detected by TDS and that apparently resulted in a low heuristic detection rate."

We did not make this mistake. On the contrary, we acknowledged that TDS has a good heuristic (compared to other scanners). Looking forward to TDS-4.

5.
@All

My personal conclusion from all our testing is: There is no perfect AV/AT scanner. Therefore, it makes sense to use several different scanners, a personal firewall and maybe a system firewall like System Safety Monitor. Such concept of layered protection in combination with common sense will make it much harder for any intruder to compromise your system.



Nautilus

 

Take care for a complete translation with all explanations and claims made so the developers involved can take note and react adequately.
In the meantime look at the thread Wayne pointed to at DSLR; further on the DCS sites (since many weeks, months even) is an interesting page about recommendations when chosing AT protection.
I prefer to see AV/AT and specialistic AT products as two complete different worlds, which products can't be tested in one and the same way.
It would be the same as telling TDS can act more or less as a basic fire wall and compare it with firewalls, which comparision is not realistic either.

I guess the name of the site/test "Scheinsicherheit" , feint or appearent security is offending and one reason more of my first reactions.

Again, submit the missed samples to all the developers so they can look at them and if necessary/possible add extra detection for those classes. It's the only way to keep an open and proper discussion.
I may remind you it it good usance in virus/trojan tests to supply vendors with the samples, or they are even send in by the vendors to a test database for this goal.
Nothing secret in this, all an even chance and only winners in the end.
Thanks in name of security for the ineternet community as a whole.

 

@Jooske

Re the malware samples: I sent a PM to you and explained why Wayne is probably not interested in compressed! zoo samples. (He has my e-mail and I am prepared to send a few samples to him. However, he already knows how to implement UPX unpacking support and can create his own samples if necessary.) We do send uncompressed malware samples to software producers. Please note that Rokop from Rokop Security was recently critized for allegedly (!) sending compressed zoo samples to software producers.

Scheinsicherheit means "illusive security". I feel that this term is not offending but adequate if you compare adverts and reality. We do not contend that every scanner is an example for illusive security.

Nautilus

 

I always liked very much how this (now old) site dealt with reviews, the integrity and done by a very knowledgeable technician knowing what he was talking about.


Samples should never been sent undemanded, of course.


Illusive, illusoir, appearent, feint security, the blinkers but no reality.

 

@Jooske ("Take care for a complete translation with all explanations and claims made so the developers involved can take note and react adequately.")

I did not complete the translation of our website, yet. But I described the upcoming test procedure so that also non-German users will know what we do.

Please see here : http://home.arcor.de/scheinsicherheit/procedure2.htm

I would be interested in your thoughts (including negative thoughts). Do you think this is a fair and useful procedure?

I would also like to invite anybody else to comment on the procedure since it would be hard to correct any mistakes after the tests have started. In particular, I would be interested in Wayne's opinion since he has already made up his mind in respect of the credibility & good pratice of AT tests ( http://www.diamondcs.com.au/index.php?page=archive&id=sec02 ).

Cheers,

Nautilus

 

Process Guard is one of our answers to many big threats. Private malware, modified malware, RETAIL malware. Hope to see a mention of this and your thoughts on it

 

@Gavin

I mentioned SSM and TPF (and failed to mention ProcessGuard) in our article about DLL trojans and elsewhere because ProcessGuard had not been officially released and tested at that time. AFAIK, the pre-release version did not support SetWindowsHook etc.

There is nothing which prevents me from updating our website after the release of PG final provided (i) I like it and (ii) I become a member of the DiamondCS reseller team

Just kidding. Of course, we will have a look at PG. But currently, our main priority is to finalize the new test archive and procedure. It's really no fun to create & test hundreds of trojan servers ...

Cheers, Nautilus