keylogger tests done with over 13 anti-spyware programs.

I have recently tested over 13 (+) different anti-spyware products* for their ability to find keyloggers and decided to post the result here for your benefit.

This is just a small test, nothing colossal in size, only 13 (+) products were tested against 35 different keyloggers.

I have no affiliations to any of the companies whose products i have tested and am receiving no compensation in any way by performing these tests and posting the results here.

I'm just a guy with a computer and do not consider myself an expert in this field of study and therefore you should take these result as no guarantee that your results with these products will coincide with my own. Though i have done my best to perform these tests in a fair and unbiased manner.

All products/ programs used were as up to date as i could get them and the full versions of each product were not always used, often a trial version was used, after all i'm not rich ya' know.

It is my hope that you will find these tests helpful in some way and will find some useful information here that will allow you a safer and more enjoyable internet experience.


* Not all products used neatly fit into the anti-spyware catagory, in fact some were anti-viruses.


Products / number of keyloggers found :

1. A Squared (free) / 2

2. Ad-Aware SE / 6

3. Antivir PE / 7

4. Bazooka spyware scanner / 9

5. Keylogger Killer 1.5 / 29

6. Norton Anti-virus / 0

7. Pest Patrol 4.4.3.24 / 15

8. Security Task Manager 1.6c / 35

9. Spybot 1.3 / 5 (4 keyloggers successfully blocked SB from starting)

10. SpyCatcher (Tenebril) 3.0 / 6

11. Spycop (free trial) / 28

12. Spyware Doctor (PC Tools) / 3

13. SpySweeper (Webroot) 3.2 / 16 (2 keyloggers successfully blocked SS from starting)

14. Who's Watching Me (free trial) / 1 (not really a fair test because trial is too limited)

15. X-Cleaner (free trial) / 6


Other Anti-spyware products were tested as well, such as Giant antispyware, but i did not complete the trials yet. May post other tests at a later date. I hope you find the tests helpful.

 

Hi x-man,

Thanks a lot for posting your results. They are quite interesting and are going to help me a lot. I certainly appreciate the time you put into the tests and your willingness to share the results on this forum. Thanks again!

Rich

 

x-man,

Thanks for your effort, I know you did this and posted the results with good intentions. To be perfectly honest, I am not real excited about a guest posting self-testing results. Why? No accountability for one thing (as in not even an email address is available). But, more importantly, it is the potential for abuse and the testing taken as serious research by our community. It's one thing to say, "X" is better than "Y" - as an opinion. It's different to lay out the applications and tell which app did what, without an ounce of testing methodology mentioned.

I should add here that I have tested security software and written academic whitepapers (many including extensive result analysis) - for years. Security has been my profession. My last job (I had to leave due to health problems) was exclusively devoted to security software testing in an academic setting. I say that only to establish that I am not just talking off the top of my head with no knowledge on the subject.

There are some basics when it comes to testing software:

All testing should be done A) By an accountable entity, B) Using standard software testing design methodologies, C) With a base benchmark system. That's just for starters.

With the above in mind, I ask: A) Who are you? You posted these "results" as a guest! With the 35/35 from one of the apps - it could very well be (and I am not saying that it is) clever spam. B) What was your methodology? Did you systematically test the applications in an equitable manner that assures complete fairness? (Though methodology is much more than this), C) Did you begin each test from an imaged baseline system - so that each keylogger was introduced to the system the exact same way? On a system in the exact same state? These are just a few of my concerns with self-testing results being posted.

A good example of self-testing with proper methodology would be Eric Howes' superb testing of anti-spyware apps Oct. 2-4. You can find the information here:
http://spywarewarrior.com/asw-test-guide.htm

X-man, I appreciate your enthusiasm in wanting to test security products. However, in the future, I think it might best that the posting of results be withheld in the Wilders Forums. Paul Wilders and LWM will have the final say on that of course. Posting in generalities about your preferences is a separate matter. "Testing" from an "unbiased" point-of-view that is posted as quasi-serious research is quite another.

By the way, you included "Who's Watching Me" - which is a program that showed lots of promise a couple of years back, but was abandoned in December of 2002. All attempts to contact Trapware and its owners have been futile. The website is still there as if it is in active development and maintenance. It is not.

Wilders is proud of the tradition of allowing anonymous postings to protect privacy. However, you sound like you would make a good addition to our community and I hope you will, at the least, consider registering.

All The Best,
John
Luv2BSecure





.

 

Well what can i say to Luv2bsecures eloquent response to my post? (applaud) Thank you for your attempts to discredit my results, but i feel people are just too clever these days to keep them in the dark on such matters. Others will (hopefully) do their own tests now and will realize that many of these so-called anti-spyware scanners really aren't all their cracked up to be. In fact i recommend you do your own tests, and you will see for yourself how well certain products work and how others don't.

If anyone feels my tests are not scientific enough, i don't have any problem with that, I never said they were tests done at a scientific level, but please don't accuse me of posting spam just because the results were not to your liking.

In fact i would love to see more honest tests done on many of these spyware scanners and i more than welcome tests done by people like Eric Howes and others, as long as they are done WITHOUT ANY behind the back dealings in any way. In other words if any of the products being tested are also paying advertisers on the web site(s) or publication(s) in which the test are being presented, then i don't feel those are the kind of tests i would trust or believe.

I have simply done these test with the idea of helping others find more useful products in the fight against keylogger type programs and am NOT suggesting that anyone goes out and buys the products i found to be more effective at finding these spying programs. Notice no websites were given, i'm leaving it up to you to find the programs if interested.

I feel there really should be more tests done in this area of keylogger detection, an area i feel is often overlooked in many anti-spyware tests. So i have just tried to help out to this end. It really does amaze me that more tests are not done in detecting and removing these programs due to the damage that could be done by having just one of these types of programs on your system. Hopefully others will step foward with more good honest tests of these products. I for one would welcome them.

 

Perhaps one way to benefit from this testing is for x-man to submit the undetected keyloggers to Ad-Aware, Pest Patrol, Spy Sweeper, SpyBot developers and see if they get incorporated in the updates by developers.

That would benefit us all if detection were incorporated.. I choose these 4 because they seem to have the highest concentration of users.

 

Thanks for taking of your time in your tests....and would like to ask you to share with us which "35 different keyloggers" you used ?

 

Well originally i was going to post all the keyloggers used, and which ones beat the different anti-spyware products, but then i decided if i had done so it would have made it too easy for someone to use this data for negative ends. So i decided against it.

But i don't see any real harm in posting every keylogger used in the tests, but simply omitting which ones beat the scanners and which didn't, if the mods will allow me to do so.

Believe me anyone can obtain the keyloggers i used in these tests and that's why the results were really so surprising, because you would think the manufacturers would have incorporated the sigs in their products by now. Or is it that many just aren't very good at finding keyloggers? I'll let you decide.

Waiting approval to post the keyloggers used in the tests.

 

I do think you have very good intention's with these tests. I own Security Task Manager, PestPatrol, Spycop & Spy Sweeper. I also use Ad-aware and A Squared (free). I must concur with siliconman01 and Bubba that it would be more useful to also post details of the Keyloggers used. Which ones each application did or didn't detect, what application did you use so that the keyloggers could record your activity. ie did you run the keylogger then open notepad, type some words and see if the anti-spyware application jumped in? Also, details of the test pc used. ie what operation system was on the pc? and most importantly, what versions & build of the keyloggers and anti-spyware applications?

As i said, your intention's seem genuine. Just with tests like these and with so many applications and keyloggers being tested, it would be useful to have a full breakdown.

Thanks,
muf

 

X-man

Good information. But you should also submit those Keyloggers that managed to breach the Anti-spywares to the respective companies so they can do something about it.

Cheers

Chew

 

As a Global Moderator....I would not have asked you to post them if it was against our TOS.

The names is all I am asking for....NOT links to keyloggers....which is against our TOS(Terms of Service)

 

x-man,

Uncalled for. I tried to explain how I appreciate your enthusiasm, etc. I also explained why I have a problem with these kinds of tests. I did not accuse you of spamming for your product that caught 35 of 35. I only said that the potential is great - especially from someone who has not registered as a member of this community.

Too tired, frankly, to discuss this one any further. Your seeing my response as "attempts to discredit" rather than what I intended is unfortunate. I thought my response was clear as to why this is of concern.

 

I can understand a little of why he/she is suspecting people are trying to discredit. A person tests some applications and posts "look my tests". The first thing people do is start saying that your test is no good because A, and because B, and also C. It tends to be a rather harsh, blunt and to a point, a brutal way of assessing someone's attempts at testing some application's. Not everyone is born blessed with the natural ability to perform a test and discover their first attempt is recognised and approved by the security community. Sometimes someone needs a little nurturing, and guidance in the right direction. Not to be shot down and treated like someone has committed a terrible sin that must be condemned by all and sundry.

I think this person could do with a little encouragement, and pointing in the right direction, maybe? After all, there is not a difinitive right way to test, only a way that the security community will feel is acceptable. Pointing to other tests that show their methodology is a great start, as is people with this sort of experience providing direct guidance here within this thread. I welcome people trying out applications. But obviously it has to be done professionally enough that it provides constructive results.

I have no doubt that some people who are regulars here have been doing this sort of thing for years, and you should be applauded for your time and effort. At the same time you should be glad that new blood is coming through and given time and hard work they may gain the experience and knowledge others here have. People have to start somewhere, and usually as a novice. This user is trying to help, whether correctly or not, they are putting time and effort in which should be encouraged. Put an arm around them and say "Here, this is how it would be better done".

muf

 

i have to agree with luv2b here. all i can say about this thread is it's interesting how some people treat computer security in a serious matter and others see it as a hobby. the hobbyists all want to do their kitchen table tests and unfortunately they get cheered on by some. bad idea.

 

Here's the list of keyloggers i used in my tests.

1. 007 keylogger
2. PC Bloodhound 1.1
3. PC spy 2.4.1
4. Actmon computer monitor v 5.11
5. Auto keylogger v 5.2
6. Pal computer surveillance system 3.2
7. Desktop spy agent
8. Blazing tools perfect keylogger lite v 2.80
9. Family keylogger v 2.80
10. Ghost keylogger v 3.80
11. Invisible keylogger 1.1
12. In the know 1.17
13. Home keylogger v 1.70
14. Key key 2000 professional 1.22
15. Keyboard logger 1.3
16. Looxee keylogger v 5.0.1.4
17. Real spy monitor build 2.13
18. Personal inspector v 400b
19. Computer monitor keylogger 1.0
20. Spy anytime pcspy 2.3
21. Sc-Keylog 2.25
22. Orvell monitoring 2004
23. Pal keylogger 1.01
24. Spyanywhere 3.01
25. Spybuddy 3.1
26. Spy-keylogger 1.0
27. Win-Spy stealth window monitor 7.1
28. XPC spy pro 2.02
29. Wintective keylogger and screen capture 2.2
30. Keylogger Express 1.01
31. Advanced Keylogger 1.0
32. Quick keylogger 2.1
33. Handy keylogger 3.24
34. NS keylogger 3.24
35. Ghost keylogger lite v 3.8

It would seem there is much talk around here, but very little action in regard to keylogger tests, at least i've never seen any good ones. That's part of the reason I did the tests in the first place. So if anyone qualified thinks they could do a better job then please do so. I hope this whole discussion will actually generate more of these tests by reliable and trustworthy individuals.

And as for the comments made by Luv2bsecure and others, thanks for your input, and I leave with no hard feelings against any. Actually I thank you for your valuable input into the matter and were circumstances reversed I would have probably responded in a similar manner. Thanks to all for your views and comments.

 

Hi all,

I have been for quite some time looking for some information - any information - on keylogger exposure. As I have experimented, I came to roughly the same conclusions that x-man's results seems to confirm. It is nice to have some confirmation - any kind of confirmation. As x-man has said, there is a paucity of information in this area. I would love to have similar information concerning Process Guard and Snoopfree. My guess is that Process Guard would perform similarly to SSM and Snoopfree would be similar to Keylogger Killer.

I would like to thank x-man for going taking the time to perform and publish the tests. It has certainly helped me in the short-term, and if shakes the trees a bit and gets vendors looking deeper into this issue, I think it will benefit all users in the long term. Thanks again x-man for helping me out.

Rich

 

Very well said x-man.

Cheers, TAS

 

Thanks Richrf, I do think my tests are valid regardless of what others may post to the contrary. I look at it this way. If you have a keylogger on your computer, set to log/run at system startup, then all your security apps, designed to do so, should detect the logger, right?

If not with their mem scans then at least with a full system scan, and if they don't then what good are they? Because in the "real world" special setups and OS's and guidelines are not used, so if your scanners are not detecting the keyloggers then i say they are not doing their job.

Most people have a wide variety of different programs and configurations and the scanners I tested should detect the spy programs regardless of what computer setup I am using and what programs I am running ect... and if they don't then what the heck are we spending money on them for?

But anyway I will try to describe the ways I used to detect the spy programs, as best I can, in my layman's terms. First I would shut down all running programs. Next install the keylogger and configure it to run/log at the next system start. Then restart Window$ (using ME). I normally have a few programs set to open at start up on the test machine- Pest Patrol, WinPatrol, NAV, Tea Timer, X-Cleaner and my firewall.

Then I would watch to see if any of the scanners would pick them up. WinPatrol would always notify me of a new program added to the start up which I would allow to continue the tests.

Also SpySweeper would do this also if set to start with Window$, but I don't consider this a detection by SpySweeper or WinPatrol because it is not an actual detection of malware and just a notice of a new program being added to the start up.

Often the names used by the keyloggers was so obscure as to be almost impossible for a novice to determine whether it was a spy program or some Window$ process (ex. SVCHOST.exe) and little if any info was given by these programs about the new start up item.

Next I would begin my manual scans with each of the programs I have listed. One at a time I would go thru full scans with the programs to see if any keylogger was detected running. After, just to double check, I would go directly to the file/folder that contained the spy program to see if a direct scan of the file/folder would detect the spy program.

After each spy program was scanned with each scanner, I would use Norton's Go-Back to restore my system back to the exact state it was in just before the first test. And would repeat this process throughout the entire series of tests.

All manner of system restarts and such was used between scans and tests and only one keylogger would be loaded and tested at a time. Also after I completed the tests I went back and tried some different startup configurations sometimes starting with just Spysweeper instead of Pest Patrol ect... So hopefully this describes the processes enough to those interested in how i came to the conclusions i did.

 

Okay, in all honesty, I was going to let this go. However, I must stand for something here or allow methodology to be run over by "everybody is a tester" thinking and make a mockery of this forum as a place that gives safe haven and warm welcomes and thank yous to amateur attempts at software testing and evaluation.

I don't believe that people here at Wilders want this to be a place where anybody can come, log-in as a guest and post self-testing evaluations. I honestly believe that is a serious mistake. Guests are welcome here. I think it's a good thing - for the sake of honoring anonymity. But that should come to a HALT when a guest begins posting tests and evaluations.

Let me address x-man on several of his points in the post above. I will try to keep it short and simple.

1. He just told you he did the testing under Windows ME. That just throws out ALL meaning for anyone using Windows XP. WinXP is built on the NT kernel and has different paths, structure, etc. Something "found" by an anti-keylogger on ME, may very well not have been found on Windows XP. Conducting "tests" on an OS that most everyone agrees is the worst Microsoft OS ever released - and is used now by less than 5% of computer users - shows why I am concerned about this kind of thing. The results mean nothing to an XP user. Nothing.

2. As to whether the tests are "valid", the question becomes valid to WHO? Polling, for example, is a science that would make a good analogy. When the GALLUP organization does a poll, using scientific methods, it has instant validity. We know Gallup's standards and methodology. Could x-man pick up his phone and call 100 people and ask a poll question and get an accurate reading on public opinion? Absolutely not. So it is with software testing and evaluation. He talks about "in the real world" that certain standards are not met. Well, that's wrong. Using scientific methodology for evaluations allow the "real world" to have confidence in certain products and services.

3. You can't test apples and oranges. Anti-Spyware/Keylogger scanning software was tested right alongside process guards/detectors.

4. The three most popular snoopware programs were not even tested! They are:
- Specter Pro
- WinWhatWhere Investigator
- SpyTech SpyAgent

The three keyloggers above account for over 50% of all keylogger sales in North America. Again, these were not even tested.

5. Who's Watching Me is a program that was abandoned in 2002. It was included in the tests. This cuts to knowledge of the subject material of which you're "testing."

The five points above, alone, show why what somebody above called, "Kitchen table testing," is not to be taken seriously. The results are the same as if I were to call 100 random people and pass the results off as true-blue "polling." The fact they were conducted on Windows ME means that they mean anything at all only to those who use Windows ME. That's testing apples and oranges again. ME is built on the 9X kernel. XP is built on the NT kernel.

There are real reasons I stand for standards in testing here. I don't mean to "flame" or be "discouraging" or anything else. Someone suggested I should be more encouraging and "guide him" in his testing and offer ideas. Again, that would be like Gallup talking to a kitchen-table "pollster" and taking the responsibility to train someone on something it took years for them to learn. It's just not that easy.

If I appear to be cranky about this, please allow me the room to stand for high standards and professionalism on Wilders. People from all over the world come to Wilders in attempts to learn about computer security and privacy. We owe it to those countless people, who do nothing more but come here and read for guidance and education, that the forum offer top-notch information. That means not encouraging "Joe or Mary Anybody" to "conduct testing" and insist that it's somehow "valid." We are either a place for professional discussion of security with higher standards to uphold than accepting anonymous research postings - or - we lose all credibility as a place that encourages research and evaluations by people who haven't a clue as to what they are doing, "just a guy with a computer." I strongly choose the former.

Best regards,
John

 

Well regardless of the test, the various Keyloggers should be submited(all vendors), you can zip all of them up and send them using the dslreports security forum one click submit link.

 

Ok Luv2bsecure, I see some of your points here. You're saying that no one but professionals should be allowed to posts software test results at Wilders. Perhaps a good idea and I wouldn't have a problem with such a rule myself seeing where there could be a mass influx of all sorts of inaccurate tests going on and that could lead to much confusion. But where are all the professional tests then? I don't see any. I was merely trying to fill a void by doing tests in an area which I felt was severely lacking. And seeing where no one else wants to take the time to do them I figured I would attempt to help out. Perhaps if the so-called professionals would get up off their backsides every once in a while and do the tests others wouldn't have to.

I find it hard to believe that you would get very different results using the two different OS's. I agree it is possible to miss a few programs, but i doubt the differences would really be that severe. The programs are generally designed to work on 9x/ME/XP machines and i would assume they should work equally well, or nearly so, on the different OS's or people are just plain getting ripped off then. I will be testing on XP to see how many differences there really are between the two OS's.

As for my not testing the 3 most popular keyloggers, so what? I just tested a variety of easily available keyloggers to show how well the anti-spy programs could detect them, that's all. My tests did not require that i use certain snoop programs over others. I just wanted to show how well some of the different available anti-keylogger type programs would detect a variety of keyloggers.

I'm not really sure why i included Who's watching me in the tests results. I tested a number of other programs that i decided not to include as well, and i guess this one just slipped between the cracks. I think everyone can safely just ignore the results for Who's watching me.

I do hope others will do their own tests though and find out for themselves how well the different anti-spy programs really work or not. That way you won't have to just take someone elses word for it. But hopefully i have helped some folks out with these tests even if not everyone agrees with them. Heck at least you may walk away with a few new anti-spy programs that may prove helpful in your battles against crimeware.

 

As this discussion hopefully continues....one of the questions that keeps coming back to me is....Should anti-spyware products be held accountable for keyloggers ?

My thinking is more along the lines of....is this an area where I want anti-spyware products to go ? I am very uncomfortable with many programs trying to bundle everything but the kitchen sink into their programs and with my limited knowledge of keyloggers....I'll ask again to those in the know........Should anti-spyware products be held accountable for keyloggers ?

 

Hi Bubba,

I am also very interested in knowing to what extent current products cover keyloggers. I think x-man's tests are at least a start. I have been searching for information everywhere and there doesn't seem to be much discussion. Is it because there are so few keyloggers? Is it because they are well-covered? I don't know.

I tested PrivacyShield Snoopfree but I have found not tests or verification of its technology. Ditto for Keylogger Killer. Anti-Keylogger from Raytown seems like a decent enough program but it is $59, and here again there is very little information. I like what I saw in x-man's test about SSM - but SSM is erratic on my machine. PG seems more stable. Maybe PG is the best answer. But maybe a keylogger will sneak through. What I lilke about the dedicated anti-keyloggers like Keylogger Killer and Privach Shield is that they positively indentify the presence of a Keylogger, if one is using the hooks that these anti-keyloggers are monitoring. But from x-man's tests, it looks like PG catches everything while the anti-keylooggers are only catching those keyloggers that use the hooks that are being monitored.

I think this is a good area for more exploration and discussion and I am sure glad x-man presented his tests. Not as a conclusion - but as a beginning for more in-depth discussion.

Rich

 

Bubba, This is a great point!! Products that don't claim to detect keyloggers should not be held accountable. It's hard to put Norton AntiVirus, process guards, and true anti-keyloggers in the same test and "match" them up. I feel that if they make the advertised claim that they detect keyloggers then they should detect keyloggers - period. Some of the anti-spyware products do it almost as an afterthought. To me, they should get on the bus or get off at the next stop if they aren't committed to it. Which is fine - do what you do and do it well, but if you can't do all you advertise - rethink the marketing.

Everyone here should realize (and I'm sure most do) that a keylogger on the PC could possibly mean financial accounts wiped-out, your identity stolen and more. As for personal security, they are the most insidious of malware.

Right now, the major products that really stake a claim on protection from keyloggers are SpyCop, Spy Sweeper, SnoopFree, Anti-Keylogger, and Keylog Killer. The others, without question, focus on garden variety spyware/adware and keyloggers are "thrown in"......

Bubba, thanks for raising probably the most important question in this entire thread.

John

 

Hi John,

I totally agree. The products that I have been looking at - and that you have listed - are all in the anti-keylogger territory. Ewido, for example (and I am just mentioning them as an example) claims to have anti-keylogging protection, but I have not idea how good it is or their commitment to the category.

All the anti-keyloggers on your list seem like good products. All seem to work as claimed and ones such as Keylogger Killer, Anti-Keylogger, and Snoopfree have all positively indentified programs that are using keyboard hooks (e.g. BOClean). Anti-Keylogger, is one that seems to be able to identify legitimate software and ignores them. Keylogger Killer and Snoopfree requires the user to "allow" execution, like Process Guard and SSM.

Given all this, besides x-man's tests, I have found nothing to go on. Anti-keylogger seems nice, but it is expensive. Do you have any further thoughts of your own or have you seen reviews anywhere that compare these products. As I said, they all seem to be doing a fine job, but without keylogger tests (like the one x-man did) and without and firm understanding of the way they monitor hooks, there really isn't much to go on at this point - as far as I can tell. Any other comments you might have would be very much appreciated. I totally agree with you, the financial risks are substantial.

Thanks,
Rich