Secunia Test of Internet Security Suites

What is in the lab today, might be in the wild tomorrow.

Detection of one exploit might be equal to etection of hundreds and thousands of malicious codes.

I will not discuss whether the exploits used by them were important or not. They started a new trend. May be some one can perform same tests in a more better way, using in the wild exploits etc.

 

and will also interfere with using the exploit for the legitimate purpose it was intended, thereby making it a false alarm...

 

Aigle, I think you have missed the crux of the test.

Secunia has NOT TESTED for vulnerability detection of systems. But has checked if AV suites can via ON-DEMAND SCAN detect EXPLOIT CODE for various vulnerabilities from within various sample files.

Thats why the tests have been blasted. As stated by Alex Eckelberry in Sunbelt Blog :

 

Ok, I am not an expert so it will be interesting to see what Norton people say about this in this case.

Anyway I am not holding my beath on this test. I just thought detection of exploits is very important for AVs.

 

IMHO, the answer lies in Patching and updating Behavioral/HIPS components to prevent exploits. Eeye excels in this.
Signature based scanning for exploit is a very costly method of solving this problem.

So the test is lopsided. If someone tests the whole package in totality for such exploit possibilities, then its worth taking notice.

 

No, because I don't just rely on my AV to protect me from such exploits. I do, however, keep my system patched and updated with newer releases of the kind of products Secunia keeps track of.

 

First: Detecting Exploits via Signature in on-demand scan is USELESS. And Secunia should know that! Of course you can add a exploit signature ( for example the shell code ) or even a "generic" signature with several known things such as a NOP 0x90 bed-in and out. However, that's not the point here. AV Vendors DO provide specific detections for this that actually WORK when the exploit is executed. With on-demand scan the exploit is NOT executed, it's just scanned like a normal worm/backdoor/trojan sample. For detecting exploits in a on-demand scan you would have to have some REAL virtualization system and not just a emulation in order to detect that. Only a very few AV vendors have that.

That said: The idea to test that isn't bad. However, how they did that was extraordinary amateurish and unprofessional.

 

Btw: Good morning Kurt

 

I've to agree with Mike and Stefan.

Just scanning a bunch of self-made exploit test files and claiming security software can't block or prevent exploits it's not really a big test.

Detecting exploits is not trivial. As Stefan said before, just detecting the vulnerability can result in a number of false positives. How do you really know if a corrupted file was intentionally corrupted to be bad or, instead, it was corrupted because of any other reason?

Then, if you want to do a better detection, you've to add a good shellcode generic detection. As Stefan said, shellcodes are often similar because virus writers are really lazy. Even doing so, sometimes this can result in a increase of system resource usage. You can't just scan every file looking for shellcode-like piece of codes, it's crazy. You could tune up the scan for specific files.

But the best thing shouldn't be basing only on signatures to have an exploit detection. There are other ways to isolate exploits, even generically, when they get executed.

Claiming 'poor exploit detection' and basing this sentence only on a signature scan it's not useful to anyone. Well, maybe only for someone.

Anyway, the positive thing - as Mike said - is the concept on which the test has been based. Exploit detection testing isn't a bad idea at all.

 

except that it ignores the legitimate use of exploits in operational security...

detecting exploits with actual malicious payloads is one thing, but cart blanche detection of exploits in general has obvious (to me) unintended consequences...

 

Having a product that tells that you are unpatched (even when it works correctly which Secunia does not in many cases), is not very useful if you dont apply the patches. Lets face it.. no one applies patches.

 

I think you are missing the point. The reason you want to detect the exploit rather than the payload with is a generic signature is to ensure you provide protection that is proactive. You dont have to keep revving the signature just because the shell-code used in the exploit changed or the script used the GUID of the activeX instead of the name.

Yes, I have played with a lot of ADODB.Stream exploit scripts and Kaspersky's protection is pretty easily bypassed. The bad guys know this (unfortunately).

 

Finally someone that understand the real impact on your security if your product cannot generically detect these exploits.. if you cannot detect the exploits, you have no shot at detecting the millions of payloads that these xploits can drop on your machine.

 

Actually you are only partly correct. Read the PDF carefully. THe web exploits were hosted on a webserver and then they browsed to them using a vulnerable browser. ON-DEMAN scans were not used for this part of the test.

ON-DEMAND scans were used for the exploits that existed in files... that I agree was a poor idea. They should have double-clicked those files using a vulnerable versio of the app.

 

Once again.. another sore AV vendor that didn't read the PDF carefully. It clearly says under Test Methodology that the web exploits were hosted on a web server and they browsed to them using a vulnerable browser with vulnerable versions of 3rd party ActiveX's installed. The security product better detect and block the exploitation of the ActiveX before it exploits the browser. The did not do an ON-DEMAND scan when testing the web-exploits..

Where they did screw up was use the on-demand scan when testing for exploits in vulnerable file formats like swf, pdf, xls etc. That was bad.

Does sunbelt have generic detection for the RDS.Dataspace exploit ?

 

I agree with regard to your web-script vulnerability test jib. The fact remains, most suites don't have a HTTP scanner. Which do are far to generic and don't delve into checking JS,VBS,SWF for exploit code.

For web based exploits sandboxing or virtualization is the only answer. Since the browser is running with admin rights in most cases and any exploit defeating the browser becomes all powerful. Whereas the browser is usually in the whitelist for AV suites, so their HIPS tend to miss the same.

Another very interesting observation is about Norman and their *sandbox* technology. If it was truly sandbox it would have protected against atleast some attacks. But it looks like Norman Sandox is just a over-rated-marketing-hyped heuristic scanner.

 

Read the PDF.. they did host the web exploits on a web server and browse to them. Its the file exploits that were statically scanned. You dont need shell code or NOP string detection to detect exploits. Check out the browser protection in NIS2009.

As an example, I challenge you to create a working RDS.Dataspace exploit that that when hosted on a webserver will bypass NIS's detection. I have not been able to create one. Just because your product sucks at generically detecting web exploits, doesn't mean you should bash the tests.

 

Why ? Because they are self made ? Thats BS. If Secunia can self-make them, so can many other hackers in the world. And they are making them.

Check out the Browser Protection in NIS2009. Browse to a webpage containing any variation of the RDS.Dataspace exploit (or any other exploit for that matter) using an unpatched XP SP2. I used RDS.Datapsce exploit since its the #1 exploit on the web. See that NIS will detect and block it. Try creating your own using <OBJECT> tags, "new" instantiation, GUIDs, encryption, VBScript whatever... NIS will detect all of them.

Test and learn. If you can come up with a JScript/VBScript that can bypass NIS2009, PM it to me. I have tried all kinds of polymorphic JScript code. If NIS2009 has a signature for it, then no amount of obfuscation will bypass it.


Then, if you want to do a better detection, you've to add a good shellcode generic detection. As Stefan said, shellcodes are often similar because virus writers are really lazy. Even doing so, sometimes this can result in a increase of system resource usage. You can't just scan every file looking for shellcode-like piece of codes, it's crazy. You could tune up the scan for specific files.

But the best thing shouldn't be basing only on signatures to have an exploit detection. There are other ways to isolate exploits, even generically, when they get executed.

Claiming 'poor exploit detection' and basing this sentence only on a signature scan it's not useful to anyone. Well, maybe only for someone.

Anyway, the positive thing - as Mike said - is the concept on which the test has been based. Exploit detection testing isn't a bad idea at all.[/QUOTE]

 

I do, don't you?

 

If its not an automatic update, then no I don't. Just dont have the time. Besides there is no way to absolutely be certain that you have patches for very single application, and ActiveX object on your machine. You could use Secunia (surprise, surprise), but its FP-prone.

So overall I would bet that even for most people on this forum, there would be at least one ActiveX or application that is still vulnerable on your machine, whether it be a Winzip ActiveX, or a RealPlayer ActiveX, or a WebThunder ActiveX (for our Chinese readers).

Ofcourse all this applies only if there IS a patch available in the first place. And we all know how long that takes.

 

True, but one can make sure they have the latest version of the application installed. It may not protect you from the newest vulnerabilities discovered since the last release of the program or module, but goes some way to protecting against previous loopholes until the next update or a patch is available.

 

Latest versions of how many applications. 20 ? 30 ? 50 ? Some of the activeX objects dont even have "installers" and certainly no installer updaters. I think its next to impossible.

 

Update on the Secunia Test

Secunia has just updated the PDF to include the details of all the Web Exploits they used for tested. Earlier they had only included the file exploits. Now all the htmls have been included as well. VERY IMPORTANTLY, to reiterate, the Web Exploits were tested by first installing the vulnerable ActiveX object, and then browsing to a web page hosting the malicious html. Thats a perfectly legit way to test drive-by downloads and most products sucked.

 

New info

http://secunia.com/blog/30/

 

An interesting point here is:

* All security suites were installed with default settings
* All security suites were tested in the same way

Wonder what the results would have been if they were tested at maximum settings.