New leader at Matousec

Night Raven: Just FWIW you're supposed to turn off HIPS before scanning with ARK tools, at least AFAIK. ARK tools are mostly incompatible with HIPS by definition.

Other than that... Well, shame about PCTools Firewall, that is annoying. I'm surprised about OA though, I figured it would do very well in a test against actual malware, and I've never encountered such bugs in it myself.

(Then again, I've never actually had the opportunity to test it against malware.)

 

Sounds like OA did its job just fine. If you blocked it the first time, the second time you shouldn't receive any warnings via pop-up. You blocked it. It won't run. If you go under the logs it should show that it tried to run but was blocked.
How did the other firewalls fare? You didn't mention anything other than PC tools and OA.

 

Hey, why didn't I think of that? *slaps forehead*

 

Strange, I've never heard that. Either way the other programs didn't give me any problems with neither MBAM, nor the anti-rootkits (RkU and GMER).

I forgot to mention that I was blocking everything without creating rules (block once). That did result in a lot of clicking but that's how I intended to do it. Since there were no rules created OA should've presented me with the very same popups it had when I first ran the malware, but it didn't. Every other program did but OA didn't. Still think it had done its job well?
Seems like some kind of a bug to me. A nasty one in my opinion if I have to be honest.

I didn't mention about the rest because I thought it wouldn't be of interest. Besides it was a pretty amateur-ish test.

Bare with me as it gets a bit chaotic at times. It's a bit late here (and I'm a bit tired), I want to get across a lot of information and english is not my native language.

CIS did quite well. Not as flawlessly as one might expect if one looks at the Matousec tests but still well enough. It did allow the creation of some files by a couple of rogues, but then again so did the rest of the programs. If one takes only the end CIS results, then they are very good. If one factors in CIS' reputation of being THE most definitive and "OMGWTF!" godlike HIPS out there, then the results weren't THAT good.
One other thing to note was that one of the samples just failed to run. I forget what the error message was but it just wouldn't start, as if it just wasn't designed to run on XP or something. To me at least this is a bad sign. While it was a piece of malware it was still a freakin' executable and should've been able to run just like any other.
Note: Sandbox was disabled for the test!
Oh, and the clipboard of my virtual machine wasn't working with CIS for some reason.

Outpost Firewall Free did OK. It did allow file creation (in C:\ and Temp folders) for some of the samples but it lacks file protection anyway. All in all it was OK.

Malware Defender did very well and managed to keep the system quite clean of malware files/folders/values/keys. Basically it was on par with CIS but without the problem with launching one of the samples and the problem with the clipboard.

Privatefirewall also did very well and was practically on par with CIS and MD as well. It did however allow one sample to put itself for automatic startup. If I'm not mistaken it was due to the fact that Privatefirewall doesn't protect the most common HKLM and HKCU "Run" keys (the ones that contain the entries that appear in the MSconfig tool), which seems very strange to me. The program monitors/protects some quite less frequently used and more obscure keys but the most common are left unguarded. Anyway, Pf did manage to keep the system pretty clean nonetheless.

Real-time Defender didn't fair that well overall and allowed a lot of files to be created. It also allowed a rootkit to get installed.

SSM Pro was somewhere in between. Judging it by its end results alone would mean not so good final score. It also did allow one sample to start automatically at system boot. However if we consider its age and lack of file protection it actually did pretty darn well. It definitely did a lot better than RTD and PC Tools Firewall Plus.

PC Tools' excuse for a security program was the worst of them all. It allowed by far the most malware samples to autostart with Windows, allowed a rootkit to get installed and was overall just plain useless.

The methodology was the following: the samples were devided in 4 groups. I launch one sample, allow it to run (duh!) and then hit block/deny (without creating any rules) until no more popups appear. If the malware hasn't terminated by itself and there aren't any popups for some time, I terminate it manually and launch it again. After I've ran all the samples from the same group I reboot to see if any of the samples have managed to slip through and set themselves for automatic run when Windows loads. Then I do a quick MBAM scan to see what files/keys/values the HIPS hadn't alerted me of and therefor I hadn't the chance to block their creation. As some of the samples were supposedly rootkits I used RkU and GMER to check whether there had been any rootkit infections.
So basically a sort of indication of how well a program has performed were the sizes of the MBAM logs after each scan. The longer the log, the more files/folders and registry values/keys had gone through. Not a very scientific approach and way of evaluating the performance but it was what I wanted to test.
The MBAM logs were the shortest when testing CIS, Mallware Defender and Privatefirewal. Well, Pf did allow that one autostart but apart from that it did a very nice job.

I ran each sample twice after I noticed the bug in OA Free, which was purely by accident. I wanted to see whether OA would do that on other samples and whether other programs would also go haywire and allow a sample to go through on its second run. It turned out that this was one single occurance - only this sample was going through on its second launch and only when OA Free was the guarding application.
Also the network componets of each program was disabled for the tests. I wanted to test only the HIPS part since I'm behind a router.

Forming a Top3 would be hard.
CIS did perform very well but it had the two strange bugs with the clipboard and one of the sample not working, and its popups were a freakin' mystery. It definitely gets the "most cryptic popups" award.
Privatefirewall also performed very well but it missed that one so obvious registry value in the 'Run' key, which basically invokes a reaction like .
Outpost Firewall Free wasn't able to produce cleaner MBAM logs but it also didn't have such notable weird problems to report.
SSM also wasn't the best performer when it comes to not allowing the malware to put files on the computer and it did allow one sample to autostart but it also can be further configured and strengthened by adding custom registry values/keys to protect. It is also the lightest of them all so it's a small plus.

Basically the Top3 would consist of Malware Defender + two of the four previously mentioned programs. Which ones probably depends on your preference and priorities.

The Bottom3 however is very easy to form (at least for me):
- OA Free - too many bugs and instabilities combined with rather high memory usage keep it from scoring higher in my book;
- RTD - this is supposed to be ProSecurity Pro with a new name and face. I don't know if RTD performs exactly as ProSecurity does but it seems to me it just doesn't cut it anymore. At least the RTD version.
- PC Tools Firewall Plus - practically a placebo. Better than nothing, yes, but still very bad. I wouldn't waste my CPU cycles on this crap.

If you read this, then you're a freakin' hero to have lasted so much and badly formulated information. You've earned your beer, now go get it.

 

Read it, but alas I don't drink beer.

I'm still very surprised re OA, and think something must have gone wrong there. OA, despite its annoying chattiness, has always struck me as being pretty solid.

Interesting regarding Pf, are you sure about that? If so it may not be so great, since malware files could be created and then execute next reboot, before the Pf interface gets loaded.

As for creation of files, IMO that's not huge as long as the the files' execution is properly intercepted and blocked. But then I'm not asking for much, I guess.

Re the errors you got, that's probably because some malware can tell it's in a VM and therefore not run. It's a trick some new malware uses, to avoid being experimented on and to fool people into letting down their sandbox-type defenses. This is one of the reasons it's generally recommended to run malware on an isolated machine rather than in a VM.

(The other being that some advanced rootkits can break out of a VM and infect the OS it runs on, if said OS is the same as the one in the VM. Or so I've heard, anyway.)

 

A glass of water then.

That makes two of us.

Unfortunately, I am.

Mostly true. Still it makes a better impression if there are fewer files created, in addition to not allowing malware to execute.

While this might be the case with some malware, it shouldn't have been the case with my sample. That specific sample did run perfectly fine with every other program. It just refused to run only while I was testing CIS. This should rule out the probability of the malware being VM-aware.

 

raven,
So when you had a pop up you blocked it and didn't make a rule? Did you have the same issue when you made a rule for it?
That is an interesting bug. Of course I always block and make a rule for it so I'm sure my security is intact. Thanks for the warning. I'd hate to go using something with a false sense. It might be time for me to start looking again.

 

Wait a minute, are you saying that if no rules are created, OA will only ask that first time and somehow never ask again for that file ?
I know that you are saying that but what I don't get is have you experienced that behaviour with EVERY file or just those samples ?
I tried myself numerous executables and it alerted me every time.
Altough I'm running OA++ , maybe , just maybe OA Free has some nasty bug, or ... sorry man, you had something very wrong with your system.

 

Would you mind testing your samples against MBAM? This to check whether the defense scope of MBAM holds any relevance against the intrusion vectors of the samples you used.

Would be nice when you checked the MBAM results with Hitma Pro, A2/PrevX free and an old (semi) competitor of MBAM like SAS

Regards Kees

 

Yes, I did.

I already said that this strange bug occured with only one specific sample. And there was nothing wrong with the testing system. It was a clean Windows XP SP3 installation with only the tested product and MBAM installed.

You mean to run all the malware samples without any real-time protection and then try to clean with MBAM? Or do mean to test against MBAM's real-time protection?

 

Shown below is the bottom segment of OA's first decision screen resulting from detecting a new install.



Please notice the options on the screenshot. If you click on "Trust this program" &/or click on "Install Mode", that's it -- you're done! OA will not bother you any more during the install.

Also please notice that OA allows the user to install in Run Safer mode (LUA) &/or automatically to create a Restore Point -- thereby protecting the user in case the process turns out to be a nasty.

 

Night Raven,

Test them against the realtime protection of MBAM. Afterwards scan with MBAM. Then use other blacklist checkers and ARK to see what's gone through (e.g. Hitman Pro, PrevX, A2 and SAS).

Because you more or less used MBAM as reference, you tested the HIPS against the scan capabilities of MBAM. This inevitably will trigger some responses on the validity of your testing method. By checking how much MBAM missis, you will 'proof' the validity of this method. When the others do not find a lot of remainders, MBAM obviously is a good choice as reference.

SO for your approach it would be inetresting and I am also curious to see how MBAM performs against the samples. When a blacklist/heuristics type of defense performs nearly as well as an HIPS, then this would bring a new perspective on the rationale of a HIPS on a Windows7 install (with UAC, DEP, ASLR, SEHOP and x64 kernel patch protection).

Thanks in advance

Kees

 

Does OA clear out all of the installer rules like MD will when you create a temporary rule or do you have to manually purge these?

 

I did the test again. I tested the programs mentioned by you: MBAM, SAS, Emsisoft Anti-Malware and Hitman Pro, all their latest available versions at the time of the test (which was an hour or so ago) and their latest definitions. MBAM had its resident protection activated.

Note: the samples weren't many, only around 30. However there was a little of almost everything: trojans, droppers, adware, rogues, rootkits.

Here is how things went down. MBAM's real-time protection managed to successfully intercept and quarantine all but 2 samples. While one of those 2 samples was doing its thing MBAM did detect and intercept a few files but as a whole the sample was active also after a reboot. Then I performed a quick MBAM scan which detected 16 threats and removed them. After another reboot I scanned with the other 3 programs:
- for Hitman Pro: default scan;
- for Emsisoft Anti-Malware: custom scan with all the checkboxes left as they are but only drive C: marked for scan;
- for SUPERAntiSpyware: full scan of drive C: only with the program's default settings.

All of them detected nothing and reported a clean system.

I have to note here that none of the programs detected a value in the registry for automatic startup (in the HKCU\...\Run key). The value was pointing to to a non-existent folder and file so it was basically harmless, but it was there. I noticed this by accident when I opened the MSconfig tool.

Edit: Rootkit Unhooker and GMER also detected nothing. Well, they shouldn't have anyway, since MBAM blocked the rootkits, but I checked just in case.

 

Thanks for the tests. MBAM has shown itself to be an outstanding anti malware application. It has been my opinion that it should run real time alongside a good AV. Recently I know a couple of folks who have gotten infected with a rogue even with top notch AVs.

Regards,
Jerry

 

Hey wait a minute Night Raven, I just thought of something re Privatefirewall... Were you using the default settings? Because the application monitor is set to "Medium" by default ( = probably not so great).

Also I'd kind of like to know what settings you used with Outpost and CIS, which are very configurable, and also (IMHO) both suffer from dodgy default settings.

Of course the above doesn't apply to PCTools, which has but one setting (on or off) and evidently might as well have none...

 

All products had their settings, that I could find, set for maximum protection.

 

Okay, thanks. Looks like Privatefirewall is basically a nonstarter then, seeing as messing with the registry and infecting on reboot is a very common malware tactic. Even with the anti-executable part enabled, something could do that easily enough, methinks...

 

Could you please PM me those files?

 

Night Raven

I can't say anything about the sample size validity, but you just have proven the validity to check the protection results of others against MBAM

It also makes one think: why bother with HIPS on a x64 Vista/Win7 * when more straight forward old technology solutions do perform well.

Thanks for testing

*) Assuming UAC, DEP, SEHOP is on

 

Night Raven,

Thank you for taking your time to test MBAM, looks like "proving" test. I have a question however - did you use the "newest" viruses available? Also I can't blame no AV but MBAM for leaving some rabbish after cleanup. Not a serious deal, many AV do this. As for /Run registry key - a virus may be named as you wish and its not the fault of any other AV program for not seeing a malware through the link in the /Run registry. There's nobody to blame but MBAM. With all of this I'm convinced with the test results. Thank you.

 

Some of the samples I had were rather old and some were very new. I'm not a professional tester. At one time I decided to keep some of the malware I come across as a malware collection, and a few days ago I suddenly had the idea to put some hips programs to the test. I wasn't aiming for anythig, it just popped in my head. I was just curious to see for myself how the HIPS programs would do against those samples. I don't claim this to be a definitive test of those HIPS programs. Just like AV-Comparatives and others, this test should be taken with a grain of salt, perhaps even more.

Whether my amateur test proved which is the best product is doubtful. What it did prove (at least for me) is that synthetic tests like Matousec's and real malware are two different things and just because a product does well in tests doesn't mean it does well "in the wild". Well, actually I already knew that, but this test made me even more sure of it.

Now to get back on topic, as we've gone astray a little.
The biggest problem with Matousec's tests is the rating system, not the tests themselves. The tests aren't anything special but they are OK overall (some are stupid and some are quite good) they could give some idea of how a program is. The problem is how the products are rated. The tests are carefully devised into groups so that some of them would reach the top and some would stay at the bottom. As I have already explained in a previous post of mine, when a program doesn't advance to the next level it automatically receives a "fail" score for each test it's not tested against, which is very wrong. One can't put any kind of a score if a program hasn't been tested. One just can't assume about these things. And even though a program hasn't been tested against some tests they do take part in forming its final score which is unfair. The only fair way to do the tests is to eliminate the groups and test all products against all the testing tools, and then present the final score.

As for me, I don't care much about Matousec's tests. I know for a fact that Malware Defender is good and I'm perfectly happy with it, especially with the few additional custom file/registry groups I've added to it.

Well, in my opinion the technology itself isn't the problem. It's the work behind the scenes that is. Blacklisting can be just as effective as a combination of HIPS+smart user, but this means that the team behind the given blacklisting program has to work very hard to keep up. That's catch. Many blacklisting product vendors don't try nearly as hard as they should. They see that average users are basically idiots and they 1) return to their blacklisting program after trying HIPS, as the latter would be too confusing and difficult for them; and/or 2) believe everything their chosen vendor tells them; and/or 3) are fanatically loyal to their vendor of choice and will continue to buy their products. If a vendor gets its money anyway, why bother trying to work hard, just a bare minimum. Marketing sells, not quality. Just add flashy buttons and some cool sounding words for some "new technology" that you've (supposedly) incorporated into your program, put some faked/out of context tests on your website, claiming you're the best or something and you're done. The average user is just too stupid and will fall for this in a second.

 

I'd replace "stupid" with "ignorance" on general principles, but yeah, other than that I basically agree.

As far as app whitelisting goes, IMHO the best mechanism for that thus far has actually been Returnil 2008's Anti Execute plugin - it lets you create a whitelist through its popup notifications or manually, and then blanket deny execution of everything not on the whitelist. Shame that Returnil removed it, it really was a great combination of user friendliness and functionality.

 

I also think that "stupid" is not the correct term. Yes, some of us are ignorant of computers and the risks associated with the internet. But not stupid.

I am sometimes bothered that some people who are have a lot of knowledge on some subject hold those who do not have much of that type knowledge in a degree of contempt.
That is a great mistake. I don't know anyone who is expert in every field or even very many.

This is not intended to be a flame against you, Night_Raven, but your choice of words was unfortunate.
I appreciate your testing, and am glad to see how the various applications performed. Thanks.

Regards,
Jerry

 

...It's said "I'm not young enough to know everything". So that's the clue?