Put your Anti-Spyware Apps to the Test!

[Bubba]

Originally Posted by aigle

What does it mean? RollbackRx or any thing else.

It means when you run any of those tests you then run their cleanup tool called TowTruck which then reverts your system to its pre-test state....Rollback. Meaning it
removes any of the executable files it was able to drop on your hard drive or any of the registry entries it was able to set.

As a side note....the Network Config Change Test is only applicable to XP or greater.

[Rivalen]

Edit; Sorry Bubba - but your to quick now and not familiar with DW.

The new DW has a Rollback function that means you can erase also the eventual traces of malware that has been deactivated by DW - without using DW Rollback you have to use an AV/AS/AT to remove those harmless deactivated remains of the malware - or you can leave them be because they dont damage your computer - only take up a tiny little space on your hard disk.

This is how I have understood it - cant explain it in tech terms - so I Rolledback afterwards to test that function. I dont think it really means a lot for this instance - since after the test Spycar had a removal function to reverse what it tried to do.

Sorry I cant explain any better.

Best Regards

[Rasheed187]

Well, actually I am very disappointed with these spycar tests, these people are completely missing the point, I mean I can test these things myself! I don´t need any apps for that. I think a test like for example the DFK Threat Simulator is more exciting than Spycar. And Spycar doesn´t even seem to give the correct results.

I do not understand why security companies are not coming up with more advanced/smarter tests, maybe because most security tools will perform poorly? This gives me stuff to think about.

And btw, I do know about sites like malware.com, but the problem is that most of the exploits do not work anymore, and it´s sometimes difficult to find out if the exploits worked or not. But IMO this is the best way to find out if your tools can protect against remote code execution attacks, the ones that we all fear.

[Bubba]

Originally Posted by Rivalen

Sorry Bubba - but your to quick now and not familiar with DW

The program DW never crossed my mind until now that you make reference to it but being unfamilar with DW is not quite true

My mistake was not seeing correctly what aigle was asking about and my explanation was more to do with how SpyCar does rollback or remove if you will what they placed on your hard drive\registry. Sorry for the oversite

[aigle]

Originally Posted by Rivalen

The new DW has a Rollback function that means you can erase also the eventual traces of malware that has been deactivated by DW - without using DW Rollback you have to use an AV/AS/AT to remove those harmless deactivated remains of the malware - or you can leave them be because they dont damage your computer - only take up a tiny little space on your hard disk.

Does this Rollback can be used as recovey sustem as well if system becomes corrupt due to any reason.
Thanks.

[Franklin]

Just an observation.

Ran the spycar tests inside Sandboxie with only ZAP running realtime.Each test stated it was succesful but no changes or warnings could be seen.

I couldn't work out what was happening until I checked out ZAP's logs.

Seems I had inadvertantly restricted "generic host process for windows32 services" the process the spycar tests are using to execute.

There are many other warnings in the log stating that generic host process was denied access communictaing with other programs.

No expert here but I'm fairly sure Generic host process is a needed services but I haven't had any probs with it being restricted.

OH well,seems I will be "googling" the rest of the night trying to work this out.

[Rasheed187]

I still think that these guys are a bunch of amateurs, I mean after all this hype they come up with these simple apps? The only thing these spycar apps do is trying to modify certain registry settings, you can do the same with a lot of other apps, you don´t need spycar for this. Or is it just me?

[Maji]

Originally Posted by Rasheed187

I still think that these guys are a bunch of amateurs, I mean after all this hype they come up with these simple apps? The only thing these spycar apps do is trying to modify certain registry settings, you can do the same with a lot of other apps, you don´t need spycar for this. Or is it just me?

It's not just you. These tests are a joke...and to prove it, I executed them on one of the OLD Compaqs we use in my university's network testing laboratory to see how a computer protected ONLY by an anti-virus would fare against these tests. Needless to say I was shocked to discover that not only did some of the tests fail to execute, but that even when they did execute properly, the changes all failed. I made sure to check for things like software restriction policies and other security policies which might be preventing these programs from making their changes, but I could not find anything of the kind. After numerous trials, it became quite clear to me how ineffective these security tests really were.

If you want to test youre security, by all means don't use these lame programs. Go with tried and true security testing software or, preferably, get a security expert to perform an audit of your system. In my case, I know several individuals who would be willing to do it for FREE.

[TNT]

Originally Posted by Rasheed187

Well, actually I am very disappointed with these spycar tests, these people are completely missing the point, I mean I can test these things myself! I don´t need any apps for that. I think a test like for example the DFK Threat Simulator is more exciting than Spycar. And Spycar doesn´t even seem to give the correct results.

I agree. The DFK Threat Simulator might not be perfect, but it sure as hell gives a lot more useful indications than these completely ridiculous tests. I ran some of them in Sandboxie and not only they give very poor indication (for a regular user) of what the threat is, but they don't even report the results correctly.

I think a test like for example the DFK Threat Simulator is more exciting than Spycar.

I'm right there on the same page with you guys over SpyCar.

Doesn't come close to Threat Simulator and some others i used in the past. In fact it reminds me more on the order of a RegTick Pro for you fellows familiar with that settings modifier.

[Rivalen]

aigle!

The Rollback in DW is - as far as I understand - there to Rollback entries in the untrusted zone.

So if I act less wisely and installs - as trusted - from a corrupt CD something that also contains malware I will not be able to Rollback that since its not in the untrusted zone/sandbox. Had I installed as untrusted under DW from the same CD I would have been able to Rollback.
How this works is dependant of wether you run DW in expert mode or ordinary mode and if you have added E: (CD)to untrusted or not - so their are some setup options.

I run DW in expert mode - hehe.

With "your" RollbacK you would be able to reverse your PC to any choosen previous Rollback copy - right. So even if you make a mistake for whatever reason - you can Rollback to a clean version.

If a make - in expert mode - a mistake that lets malware into the trusted zone - I cannot DW Rollback that. If I run in ordinary mode and have say A: and D: as untrusted any installed file should also be untrusted and be able to be DW Rollbacked.

Sorry I cant explain it better - this is how I understand DW (until corrected) maybe if you read at their site you get better answers.

Edit; my explanation sounds like Defensewall is a complicated software - its not - its so easy to use - trying to understand it might not be necessary?

Said if Spycar is a poor test - I ll test that other one that was said to be harder and see if there is a thread for exchange of test results from that test.

Best Regards

[aigle]

thanks!

[Rivalen]

Where can I find a working DFK Threat Simulator download link?

Best Regards

[Franklin]

http://www.morgud.com/interests/secu...-simulator.asp

[Rivalen]

What am I doing wrong. Dont get adownload at that link. Just send me round in circles. Have you tried it?

Best Regards

[Franklin]

At the bottom of the page.

DFK-Threat-Simulator.zip (zip password: morgud.com)

[Rivalen]

I will have to wait until I get that DFK-link to work. I have heard that DefenseWall passes the test but I want to try myself.

Anybody knows of any other such malware test that is considered to be worth the effort.

Best Regards

[edskoudis]

Ed Skoudis here...

Thank you for your provocative comments. To help clarify the motivation of Spycar and its value, I’ve prepared the following responses to particular issues described in this thread. I'll preface each point made earlier in the thread with a *, followed by my response.

* But, Spycar Only Changes Registry Keys

As you know, Windows is controlled to a massive extent by the Registry. To plant itself on a system, a lot of spyware diddles with various Registry keys, including some of the ones we modeled in Spycar. Some anti-spyware tools try to prevent changes to these keys with their behavior-based defenses. Spycar tries to verify this protection by changing the same Registry keys as the spyware.

Furthermore, not all of Spycar focuses on Registry keys. The alter hosts file element appends an entry to the hosts file itself.

And, finally, please note (as we say at the Spycar website) that we released only the first batch of Spycar modules on Friday, May 5. Call it Spycar 1.0 if you’d like. We've got several other modules up our sleeve, and we have implemented them. The harder part of a tool like Spycar is to roll back the changes in a consistent and comprehensive manner. We're working on implementing those clean-ups in TowTruck and releasing the new modules in the coming weeks. Some of the new modules we're working on include:
- A simple keystroke logger, which will gather just 3 keystrokes (that would not be a mere registry change)
- Importing a code-signing cert into IE
- Importing an SSL cert into IE
- Firefox behavior alteration tools, akin to our current IE suite
- Many others...

* Spycar is Simplistic

It has been pointed out that larger, more complicated applications can test more functionality and model more behavior. But, with the goals of the Spycar project, small and simple beautiful, for several reasons.

First, we wanted anyone (not just technical specialists) to be able to evaluate their anti-spyware tool. Technical experts are welcomed to use the tool. Many have, and have provided highly useful input. But, we also wanted non-experts to be able to give it a spin and evaluate their protection.

Second, in the case of what Spycar is trying to measure, technically speaking, small and simple are desirable. If Spycar were a big, monolithic application testing a whole bunch of items in a single executable, an anti-spyware tool might detect it early in its testing cycle and shut down the testing process. Then, all tests after that would not be accurate. Serious anti-spyware heuristic testing must be atomic if it is to get results from which conclusions can be drawn. Do you let me do this? No... Well, do you let another form of me do that? Yes...

It is important to note that an all-in-one application can test whether a given application is ranking up a score of maliciousness (assigning points to each behavior before deciding to pull the trigger on an application), and shut it down when its score exceeds a threshold. Spycar does not perform that sort of testing, focusing instead on each behavior with a simple question: do you warn me about a process making this change, do you block it, or do you just let it slide by?

And finally, when considering the simplicity of Spycar, consider the EICAR anti-virus test file. Now, there is simplicity for you, and it has provided significant value in verifying anti-virus programs. Spycar is not an exhaustive test (although it has found some interesting results… see below for descriptions of some interesting findings with some vendors), but focuses on modeling certain aspects of spyware behavior.

* The Guys Who Created Spycare Are Amateurs

I cannot comment authoritatively on who is a pro and who is an amateur. Such a conversation would spread more heat than light. But, just to kick in a few thoughts: I've been doing information security product testing for large-scale organizations for over ten years, including crypto products, anti-virus tools, firewalls, IPS products, and anti-spyware tools, for organizations including telcos, banks, government agencies, energy companies, etc. Some of my public test results are located at the following places:

- Anti-virus product testing (June 2004): http://infosecuritymag.techtarget.co...art803,00.html
- Anti-virus support testing (October 2004): http://infosecuritymag.techtarget.co...rt1005,00.html
- Network-based IPS testing (November 2005): http://informationsecurity.techtarge...137922,00.html
- Anti-spyware testing, using Spycar, as well as several other methods for evaluation (May 2006): http://informationsecurity.techtarge...184258,00.html

Putting that aside, consider some of these results we learned with Spycar testing:

One of the major anti-spyware vendors (McAfee) offered no protection for Run, RunOnce, and RunOnceEx reg keys if the process that is changing them has a name greater than 15 characters in length. Their behavior-based protection worked just great unless the process doing the attack had such a name, when such protection would vanish. We discovered this using Spycar, informed the vendor responsibly, and they released a fix within 48 hours.

Another anti-spyware vendor, Webroot, protects Run and RunOnce, but does not properly protect the RunOnceEx registry keys. That's because the structure of successfully written RunOnceEx keys differ from their brethren, a fact not widely known. Again, we discovered this using Spycar, and informed the vendor.

* I Can Do Those Tests By Hand

Sure, you can, for the most part. Of course. But, few people choose to do so (see the findings for the various run registry keys above.) We wanted a test suite that was accessible to lots of testers. The pros can do their own thing. Have at it. By the way, for those who say they can do the tests by hand… have you published your results yet? Please let us know where we can see your hand-based testing results. We’d love to learn from you, and incorporate some of your testing concepts into Spycar.

The reason I said, “for the most part” above is that there are some changes you cannot really test by hand in the same way as Spycar. For example, note the process name greater than 15 character issue mentioned above. If you were to try to make that change by hand, the McAfee tool would block it, because the process making the change would be interpreted as explorer.exe, the Windows GUI. That name is less than 15 characters, so you appear to have protection when you do it by hand. Only with a separate application could you make such a change and verify the protection of the product. Yes, it is something we stumbled upon accidentally in our testing (that happens often, in the testing business). But, it is a significant result, and something that testing by hand would not have uncovered.

* Spycar Didn’t Make the Changes to My Unprotected System

One poster here mentioned that Spycar didn’t make any changes to a system that was unprotected (an old Compaq system). That’s a fascinating finding. Any idea why? Can you either send us a description of the build so we can figure out why, or run RegMon and see where it is getting hung up? As many have pointed out, these tests are very straightforward, so their failure on your box is an interesting outcome. I’d love to know why, but cannot discern from the sketchy details in your post. We’ve had many hundreds of people run Spycar successfully, so your results are a fascinating outlier.

* Spycar was Overhyped

Spycar does no more and no less than we promised up front. In all of our interactions with people, we explained as clearly as we could what Spycar would do. We got a tremendous amount of positive feedback up front, from very large software companies that I cannot name here, about the idea. Since its release, we've gotten a lot of enthusiastic e-mails from both individual consumers and IT professionals who have said they were shocked at the lack of protection they have on their machines. In the end, that's why we released it... so people could test their protection and see if it matched their assumptions.

If you have other questions or comments about Spycar, please do let us know.

Thanks for the input and challenging points—
--Ed Skoudis
Senior Security Analyst
Intelguardians

[ronjor]

Welcome to Wilders Ed and thank you for your post.

[edskoudis]

Thank you, Ron. It's good to be here.

[Rivalen]

Thanks Ed Skoudis - interesting reading - any eta on these new expanded tests?

Franklin! - can you download from that link you gave me? I ran iexplore as trusted by Defensewall but I simply cant download.

Best Regards

[beetlejuice69]

Good read Ed and thanks...oh and welcome.

[Devil's Advocate]

Originally Posted by Rasheed187

I still think that these guys are a bunch of amateurs, I mean after all this hype they come up with these simple apps?

Originally Posted by edskoudis

I cannot comment authoritatively on who is a pro and who is an amateur.

Heh so modest.

Such a conversation would spread more heat than light. But, just to kick in a few thoughts: I've been doing information security product testing for large-scale organizations for over ten years, including crypto products, anti-virus tools, firewalls, IPS products, and anti-spyware tools, for organizations including telcos, banks, government agencies, energy companies, etc.

Hi Ed, no need to list your list of credentials, you are well known or should be, for people who really are in the know. Considering the known abilites of the poster who called you an ameteur, it's pretty hilarous I think.

PS I enjoyed reading your 'Counterhack' books .*Back to lurk mode.*

[JimIT]

Hi Ed! Cool to see you in these parts!!

[Rasheed187]

OK thanks for the feedback, I understand it all better now. My comments were based upon the facts that I had expected a bit more advanced tests as you have noticed. That´s why I was not impressed at all, and when I called you guys amateurs, I meant that guys with your background should have come up with something better. But I see that you´re coming up with more interesting stuff, nice to know. Also nice to see that you´ve actually discovered flaws in certain products.

But yes it´s true, a lot of anti spyware apps do not offer strong real time protection and even more advanced HIPS can not always correctly detect certain (possible malicious) changes made to a system. I´ve tested this with all kind of applications (including registry tweakers, startup control, process/service/driver tools etc.).

@ DA

Nice to see you back, I´m surprised that you don´t have anything negative to say about these tests, after all you wasn´t too impressed with other more advanced tests. Please post more often we really need more posts from "experts" like you. And thanks for providing me with so much fun during our little private conversation via PM´s. But I see you have finally decided to take my advice, I hope you sleep better now, kuddos!