AV Testing & Packers

[Diver]

This is just my theory, it needs testing or more information:

I am thinking that the AV's with the highest percentage of detection either have very good packer (unpacking) support or else simply treat all packed executables as positive. The former is an accomplishment, the later is unacceptable in my view (and of some of the experts) but others differ.

The usual way AV's are tested is an on demand scan of a partition or drive loaded with malware. Usually nothing is archived, but if an executable is packed it must be dealt with somehow.

Some good AV's will miss a few of these packed executables thus lowering their score from 98-99% into the mid 90's. However, if the missed packed executable files were run, it is possible that the AV would recognize the malware and detect it before any damage is done.

This test is not done ordinarily as it would be difficult to automate and further test that detection prevented infection.

Thus some AV's with limited unpacking abilities may be better than they look at first.

Just my theory. Anyone got something to say about this.

[C.S.J]

drweb is an unpacking monster, and its certainly not the best detection.

something to think about

[solcroft]

Originally Posted by C.S.J

drweb is an unpacking monster, and its certainly not the best detection.

something to think about

Well, unpacking is obviously not the only factor in detection rates, it also depends on whether you have a signature to detect the stuff you've unpacked...

I believe Andreas Marx mentioned something about NOD32 and DrWeb having the best database size to detection rate ratio, this could have something to do with it...

Originally Posted by Diver

Some good AV's will miss a few of these packed executables thus lowering their score from 98-99% into the mid 90's. However, if the missed packed executable files were run, it is possible that the AV would recognize the malware and detect it before any damage is done.

AFAIK this is not possible unless the resident and on-demand scanners use different settings or scanning engines. What COULD happen is that the resident scanner will catch files dropped by the malware, which doesn't always (completely) stop a malware from delivering its payload.

Some packer detection is necessary nowadays IMHO. Some things just cannot be unpacked by a computer program, or would require extraordinary amounts of system resources to accomplish - NOD32 as an example typically freezes up for ~10 seconds or worse when it runs across complex and compound packers like Themida or NSAnti, and then it might not even succeed in unpacking it. However, when one starts flagging everything in sight as TR/Crypt.xxx.Gen, for instance...

(Come on, Stefan. I want to believe I'm wrong. )

[RejZoR]

Database size in terms of KB/MB is totally useless comparison imo. Database size in number of entries is more relevant. Some have extreme number of entries, yet they detect less than some with less entries. Size in megabytes tells nothing, company just uses different format, maybe they don't even compress it later or something else. Bunch of factors.

[Londonbeat]

Originally Posted by solcroft

However, when one starts flagging everything in sight as TR/Crypt.xxx.Gen, for instance...

(Come on, Stefan. I want to believe I'm wrong. )

I'm no expert, but Antivir's level of false positives is low considering the extent of it's heuristic detection level. I think Stefan has said somewhere here before that generic/heuristic detections of Antivir are not soley based on packers, there are always other checks made on the file before it's flagged.

[Tweakie]

Originally Posted by Diver

However, if the missed packed executable files were run, it is possible that the AV would recognize the malware and detect it before any damage is done.

Actually, there is no reason for an antovirus to detect a packed malware when it is run if it did not detect it before. More precisely: the ratio "detected when run"/"detected on demand" should be the same for packed and unpacked malware.

Of course, you may argue about those "memory scanners".

1) How do they work ? To perform their scan, they have to break the execution of the malware after it has unpacked itself and before it started its malicious actions. The problem is quite similar to generic unpacking by emulation (you have to find the OEP) but the fact that the malware gets executed, and that consequently you do not fully control the execution environment. Hence a great chance to miss the OEP and to let the malware execute completely.

2) Who is really using that approach ? AFAIK none of the main antiviruses.

Originally Posted by solcroft

However, when one starts flagging everything in sight as TR/Crypt.xxx.Gen, for instance...

(Come on, Stefan. I want to believe I'm wrong. )

To come back to the discussion in the other thread:
- AV detection is indeed an automated classification problem. It has to classify all the executables it scans in two groups: goodware/malware
- As for any classification/clustering algorithm, the purpose is to minimize the error rate. There are two kindsof errors: false positive (goodware classified as malware) and false negative (malware classified as goodware). Each error does not have the same weight: for false negatives, the weight should be related to the prevalence and/or to the dangerosity of the sample (i.e. the risk). For false positives, it is also related to the prevalence and to the importance of the executable (i.e. executables that are vital to the OS must absolutely be classified correctly).
- There is a "grey area", composed of cracks, cracked software, software that use packers that have been concieved specifically to avoid detection by AV software***, etc. Some users (e.g. my employer) would like it to be classified as malware whereas others (you) consider it as goodware. A sensible position for an AV company is to assign them a weight of zero in the classification system. In other words: ignore them, or consider that they are always well classified.
- There is a "fool area" of files written exclusively to trigger "false positives". These should be ignored as well.

I believe that's the way most AV companies consider the problem. I see no problem with it.

*** There is no reason to tolerate the use of those pakers. There are plenty of good, legitimate packers around.

Originally Posted by RejZoR

Database size in terms of KB/MB is totally useless comparison imo. Database size in number of entries is more relevant.

IMHO both are irrelevant: it is not clear how the companies count the number of entries either. Size does not matter

[Diver]

Well, It looks like several of you do focus on the difficulty of determining if the malware has done its damage if detected during the unpacking stage. My approach to that in the past is to restore from an image backup rather than to take a chance.

The "gray zone" is definitely a matter of interpretation. More stuff will fall into this area with an enterprise AV as this would include programs used for exploits.

Perhaps I should ask him myself, but who is Stefan? Is he an AV developer/researcher? Is this the same fellow who was connected with CHX-1?

[Tweakie]

Originally Posted by Diver

Well, It looks like several of you do focus on the difficulty of determining if the malware has done its damage if detected during the unpacking stage. My approach to that in the past is to restore from an image backup rather than to take a chance.

Beside all practical issues related to backups, malicious actions could include sending all your passwords to a remote computer, exploiting vulnerable computers on the LAN, sending spams or worms, etc. All this damage cannot be restored from a backup.

The "gray zone" is definitely a matter of interpretation. More stuff will fall into this area with an enterprise AV as this would include programs used for exploits.

Right. That's the "potentially unwanted programs" or "hacktools" category of some AVs.

Perhaps I should ask him myself, but who is Stefan? Is he an AV developer/researcher? Is this the same fellow who was connected with CHX-1?

Stefan is a developer at Avira GMBH, he developped Antivir heuristics.

[Alphalutra1]

Originally Posted by Diver

Perhaps I should ask him myself, but who is Stefan? Is he an AV developer/researcher? Is this the same fellow who was connected with CHX-1?

He isn't the same guy that worked for CHX-I, that was another Stefan (his name was Rares Stefan), and he has dropped off the map, same with his product unfortunately (I have copies, I just wished they had continued development and all)

Cheers,

Alphalutra1