DiamondCS - Are you aware of this site/test results ?

It seems that 235 samples from 556 are not detected, which (if my dodgy maths serves me correctly) means that less than half were detected.

http://boardadmin.funpic.de/viewtopic.php?t=22&sid=594188ac138db65ddcd691b1162b4339

 

...but make sure to read the whole site (http://scheinsicherheit.funpic.de/introduction.htm or http://scheinsicherheit.funpic.de/einleitung.htm in german with more information; a (german) full review of tds (that is a wee bit older than the forum posting you referred to) is here:http://scheinsicherheit.funpic.de/tds.htm). I have just recommended it on the forum over there. A pity that still a lot of information is available only in german, but try to read and understand as much as you can.

The thing with the test (that is one of the rare sites that gives much detail about how they're testing: http://scheinsicherheit.funpic.de/procedure2.htm)is this: It's meant to find out, how the inner workings of a scanner are working, i.e. whether or not it uses heuristics, an unpacking engine, weak or strong signatures etc. Therefore, the author has handcrafted some 550ish trojans, to trigger some architectural specifics. Normally, you would call this a zoo test, and one with an extremely small set of samples at that, too. But while the ideal scanner would of course detect everything, the scans are somehow "meant to" fail, namely in a way that lets you infer certain things about the program. The test is not meant to representative in regards to the total coverage or even quality of the scanner in question.

Nautilus there found out that a) TDS-3 doesn't use an unpacking engine, b) uses mostly strong, and a few weak signatures, c) has a very very good heuristics, and d) has a very very good memory scanner - which is a bit slow and needs some extra handling to cover modules loaded into processes. (a) will probably have to be re-evaluated when TDS-4 comes out. There's not yet any information about how DCS are going to deal with decryption/unpacking, but they are surely planning to do it.)

BTW, TDS-3 is the reference scanner over there, and on the forum you've mentioned, compare TDS's 235 misses to those of the competitors. Except for BOCLean and TrojanHunter, which are short behind, TDS-3 leads the pack with quite a distance.
(Then again, I've just noticed that you seem to have digged that site and the forum rather well for you posted more about it in the "moosoft cleaner more effective...?" thread. So, this is not meant to tutor you, but in order to have more information about the test visible at first glance already than just x out of y misses, how is the vendor responding?)

Cheers,
Andreas

 

Nautilus was simply testing to see which files are detected when they're modified to become new programs which is effectively what they are, but there are an infinite number of ways to modify a file so that it's no longer detected by scanners. As one of the easiest examples, it's well known in trojan author circles that one of the more popular anti-trojan scanners (not TDS3) can be defeated simply by appending 1% more data to the end of the file (you can do this simply by using a hex editor), so for example if a detected exe file is 10000 bytes then add another 1000 to the end and it'll no longer be detected. In this case the program itself is still identical (ie. not packed or encrypted), but because of the increased filesize the scanner ends up looking in wrong places for signatures as the signature positions are relative to the filesize. Just one simple example of modifying a file to prevent detection. Packing is another. Encrypting is another. Header modification is another. I could go on...

However, for a trojan to be dangerous it has to actually execute - if it just sits on your disk without running then there's no immediate danger as it can't do anything. When a trojan DOES execute, different scanning techniques can be employed. Even process memory scanning techniques can be thwarted but it's a bit trickier, and detection is also a lot easier (as the original program has been unpacked, etc). It'd be interesting if Nautilus did such a test but it'd be very time consuming for him.

 

So, Wayne, what do you think of the approach in general and of his assessment of TDS in particular?
And in how close contact are you with him concerning improvements in TDS-4?

Andreas

 

With all due respect to Nautilus, at the end of the day any file that's detected by a scanner (packed or not) can be modified to the point where it's no longer detected, so just testing with certain packers is quite useless - even if the file is packed and a scanner unpacks and detects the packed program inside it can still easily be modified by even the most amateur of trojan users. Trojan authors/users will always have the upper-hand over scanners, that's just a fact and you don't need tests to prove that.

We're not in contact with him.

Regards,
Wayne

 

I see your point. Still I do think that there is something to be drawn from his tests:

(I don't think that this is what his tests are supposed to prove. Rather he wants AT/AV developers to make use of certain techniques to counter popular trojan authors' strategies. By proving that the techniques do have an effect and that some have implemented them, and by asking if others are implementing them or are not planning on doing so.)

So, let's call it a fact about the principles of "trojanity" (?). As you said, "at the end of the day". But when it comes to everyday practice, what his test are proving - I suppose - is the difference in sophistication that the different scanners on the market are using. In practice some packers/crypters or other modification methods are more common than others, so covering these reduces the probability of being infected (but we all know that probabilities are not telling us anything about whether we will be infected or not, you would be right to object).

What his tests are proving - I suppose - is that scanners in practice have to become more generic. Both by relying on good heuristics and multiple scan approches/methods in addition to signatures, and by featuring a generic unpacking engine or some method by which the heuristics and other scans can be run against a file before it is actually executed when the file is modified in a way that prevents it from being detected both by signatures (because it's modified) and by heuristics (because it's encrypted/compressed).

TDS is outstanding in the first area as it is right now already. DCS are developing other tools to wing its Trojan Scanner, for, as you said, scanners in general are on the weaker side of the trojan/scanner war in principle. I am rather confident that TDS-4 will have a reasonable response covering the second area as well.

So while in principle the tests do neither prove anything new about "principles of trojanity", nor about the overall coverage of any specific scanner product, your development history somehow follows the same reasoning, and his tests are useful to make people aware of that very reasoning.
(I am looking forward to TDS-4 and to Nautilus's reaction on it and on ProcessGuard. From his arguments I should be somewhat expecting him to be rather enthusiastic about both of them. He seems at least somewhat more open-minded than many reviewers, both in the sense of not being too biased and of being ready to explain many details of his arguments.)

Just my 0.02 € - and I didn't mean to inflate/prolong this thread beyond reason.

Andreas

 

He did express good feelings towards ProcessGuard in some other posts, too busy to find them at this time

Part of what Wayne was indicating is that scanners are always going to be vulnerable in some way. We will be detecting as much as we can, and TDS-4 is way, way more advanced - but we will never expect to detect everything. But yes, we have many things up our sleeve..

The obvious answer is ProcessGuard is part of the layered defence for a reason, to stop unknown executions, DLL injection, process modification and termination, and rootkit driver installs. A little common sense is better than most scanners.

Edit-
Note that nearly ALL samples are compressed/crypted variants in which cases memory scanning comes in (just had time to browse the results now)

 

Now that you mention it. - In one instance (of the full review) TDS's could theoretically detect the trojan with the memory scan, but wouldn't launch. ntl gives some hints as to how this process killing of the malware could by sidestepped (i.e. renaming tds-3.exe), but I was wondering if I got it right, that a plain TDS-3 protected by ProcessGuard would be able to launch regardless of the malwares attempts to block it (i know it wouldn't be able to be terminated, but would it be able to launch? are those different things?) and would then detect the samples in memory scan?

One other thing - but i suppose that's something I should better ask ntl himself - is that he reports injected dlls to be detected only when you go via process list/show modulese/scan modules, but not in the default or even start-up scans. Any comment?

Andreas

 

Protect TDS-3 with PG and no "killer" current can kill it (or feasible future killer) so yes detection would be no problems for a lot of samples

There is 1 unconventional killer that I know of which actually infects explorer.exe and prevents AV's AT's and FW's (TDS included) from running. This special killer doesnt use termination - it actually injects into explorer.exe and prevents the execution of certain named EXE's instead of allowing them to run then killing them. ProcessGuard does of course block the infection/injection of explorer.exe altogether so the killer never actually works either on ANY process whether protected or not

 

that was what i was aware of already

that's what I was afraid about. "preventing the launch without needing to kill"..

That's what the question was about and what i didn't really think of. Good point, I'm convinced. I was a PG fan before, but still this is great news. I think it's good to have been made explicit.

CU,
Andreas

 

1.
It's interesting that our (partially outdated) reports suddenly give rise to an intense discussion.

2.
I cannot comment on every statement but will confine myself to highlight certain key aspects. (Additional information/answers can be found on our website.)

3.
Please note that TDS-3 missed 235 samples because only the file scanner (and not the memory scanner) was used. This is expressly mentioned in the log.

4.
We mainly use zoo samples. But most of our zoo samples closely resemble the samples used by the real bad guys. Maybe a trojan user will compress Optix with PECompact while we have compressed Theef with tElock. Maybe a trojan user will add 1500 bytes to a sample while we have added 1400 bytes to another sample. In my opinion, such differences do not really matter. A good scanner should not be affected by such simple tricks.

5.
We have never purported that BOClean, TDS or Trojan Hunter is the best AT. All three products feature a memory scanner and, therefore, provide "added value". All three products are far from perfect. The same applies to newcomer ESS Plus from ewido which is - AFAIK - the fourth AT on the market which features a memory scanner.

6.
"It'd be interesting if Nautilus did such a test but it'd be very time consuming for him."

We did such a test in respect of BOClean. We executed more than 500 malware samples in order to determine whether they can be detected by BOClean's mem scanner. This test was repeated several times. With respect to TDS-3, we have not performed such a lengthy test so far. However, we have already made many, many spot tests. For example, we made tests with Armadillo Copy-Mem II and with compressed DLL trojans ...

6.
"Trojan authors/users will always have the upper-hand over scanners, that's just a fact and you don't need tests to prove that."

This sounds a little bit too pessimistic to me. Frequently, the technical skills of trojan users (or even authors) are quite limited. If an AT features a mem scanner and uses strong signatures many trojan users will be unable to circumvent the scanner.

Moreover, small AT software developers should look forward and be innovative since this is the only way to compete with the large AV companies. I believe that the future of malware detection lies with the behaviour-based analysis of trojans and backdoors. By contrast, signature scanning is hardly sophisticated enough in order to effectively deal with non-replicating malware.

I am somewhat surprised that there is still no AT which can (i) stop any ordinary DLL trojan with the help of a "create remote thread" blocker and (ii) detect any ordinary non-reverse trojan (which is not using a visible window) with the help of port-to-process mapping in connection with a z-buffer check.

 

Addendum:

"how close contact are you with him concerning improvements in TDS-4?

We're not in contact with him."

This is 100% correct. The statements contained in our TDS-3 report as of 14 December 2002 are completely outdated. We have no idea whether the information provided to us at that time is still correct. In particular, we do not know whether, when and in which from TDS-4 will be released.

 

Nautilus, thanks for your thoughts and feedback. Anyway, back to work.