AV-Comparatives Firewall Test 03/2014

@vojta: it was a real machine. Actually, we tried it even on several machines.
Those tests were so basic that anyone of you could simply try it out and verify instead of making wild assumptions.

 

from what i gather they just installed the apps, OK-ing thru any prompts or suggestions during the install w/o consideration of consequences.

Outpost Firewall & Security Suite:
on installation, for the people who do not want to spend time setting up rules or answering the rules wizard prompts, it asks if you want to run in autolearn mode for a week to build up a rule set to cover you commonly used apps. you also have the option NOT to do this. i do not like that as if you do not read the instructions and turn off the autrolearn, it will stay in that mode too long. agnitum apparently bowed to pressure from the install it and forget it crowd who complained about having to train outpost by answering a lot of rules prompts & made it a default. hopefully this so-called 'test' will prompt them to clarify any auto-learn attempts initiated by the end user. with just a modicum of configuration, outpost should pass all these tests (to be fair, so should comodo).

additionally, if they tested by installing and then un-installing apps on the same machines and proceeding to the next app, there may have been leftovers, dlls and other driver files left behind by some uninstallers, even those that say they remove all traces. a/v & firewall progs that install at the lowest levels of the OS are prone to this. the result can be that the next app installed has major problems.
agnitum had to write a special uninstallation program 'clean.exe' that resides in the programs file folder and which needs to be run in safe mode to clear all the security settings and remove all traces. the only realistic way they could test is to format the hard disks between tests and install from a clean image of windows that had never had an a/v or firewall installed. there is no mention of doing this that i recall. i wonder what they did do. also i wonder if windows firewall was left on or turned off. having two (or more) firewalls or /v progs active may conflict.

as a/v comparitives ran this on multiple machines, what kind of network was it on? most personal firewalls are not designed to be installed at default settings on a microsoft commercial active directory lan system that may have a number of segments, subnets and associated routers & load balancing gear, etc.

 

@kronckew : we always start with fresh systems - so there are no leftovers etc.; thats a standard best-practice. Regarding windows firewall: this is mentioned in the report.

 

@IBK, good. as you note that is the only way to start. as i noted agnitum should clarify and use more sensible default installation configs, and end users should do a bit more configuration beyond defaults for their own possibly unique configurations.

p.s. - outpost users can visit us at the link in my signature if they have any installation and/or configuration questions, we'd be glad to help if we can.

also, we are not agnitum, or employees of theirs. we are a user run and administered support forum.

 

It would be nice to know why Online Armor failed this simple test since I use OA. I don't have remote management enabled on any of my machines so that would not be an issue to begin with, but I still don't like the fact that OA failed such a simple test. Fabian from Emsisoft said he could not reproduce the results of the test, and he is a security expert so I don't know what to think.

 

Second that, still waiting for Fabian to get back on this.

/E

 

I don't think you will be hearing back from him anytime soon. He said Emsisoft has him busy finishing EIS so he want be able to get back to this until EIS is released.

 

but what if you need to have open/forwarded ports on your router for games or p2p apps etc?

 

I have a question for people using windows firewall. How do you stop programs piggy backing on other programs that is allowed to connect out? because windows firewall can't block that. run the PC Flank leak test and you will see.

 

Usually games, P2P and others don't use ports that are reserved for ping, sharing and RDP. Router would still protect you against those tests.

 

but it would still be a security hole if there were open ports on your router wouldn't it? A hacker scans your ip and 99 percent of your ports are stealthed except he finds 1 or 2 ports open. who knows what the hacker can do next once he finds open ports on your router, so its now up to your 3rd party firewall on your desktop to block any further unwanted activities.

 

He can't do much if there is nothing listening on that port. He can only exploit P2P program or game and there is not much 3rd party firewall can do here (without HIPS and other non-FW components).

 

You could use a firewall that performs well in leaktests.

 

Thanks for the info!

~ Removed Remarks - JRViejo ~

 

Do you know if Online Armor incorrectly trusted the network? I'm making some suggestions over at Emsisoft because the behavior i'm seeing when it detects my network adapter is confusing to me. OA gives an option to tick a box to trust the network once your network adapter is detected, but if you do not tick the box the network appears highlighted in green even though it is not trusted. If you trust the network after initially choosing not to, and then go back to not trusting the network again the network then appears highlighted in red. I have to ask myself why was the network initially highlighted in green when I chose not to trust it, and now the network is highlighted in red after not trusting it again. This is the behavior i'm seeing on my machine anyways. I'm going to make some suggestions to Emsisoft, but I know they will not get to making any changes until their new Security Suite, and EAM is released.

Edited: 5/2 @1205 am. I just got done conducting some test of my own, and the network is actually initially highlighted in yellow when I do not trust it. I was wrong. It was the network address that was highlighted in green. I guess that just indicated the network is active. The network description, and trust field are both highlighted in yellow. The weird thing is though if I trust the network, and then go back to not trusting it again it appears highlighted in red instead of yellow. I don't understand why there is a change. I guess I need to ask Emsisoft about this. I don't understand why it would be highlighted in yellow the first time I did not trust it, and then be highlighted in red when not trusting it again. I'm sure Andrew, or Fabian can clear this up for me. I think it would be best to give at least two options when OA discovers the network giving two different boxes to tick. Option 1: Home Network (trusted), Option 2: Public Network (not trusted). I will ask Andrew, and Fabian if they will consider doing this. It will have to wait until tomorrow though. I do not have time to do it tonight.

 

Btw.. I tried to watch the video of the test conducted for OA, but it does not exist. I guess it was removed.

 

FWIW: Doesn't AV Comparatives ask that a link to its main site be given rather than a specific link to the study results?

 

Or a 3rd party anti-executable, especially one that supports .dll control, to augment Windows firewall.

 

Exactly!

That is stunning failure of methodology and execution on the part of AV comparatives that boarders on incompetence. It is absolutely inexcusable for a professional testing organization to screw up in this manner. No wonder Commodo refused to participate.

Makes me wonder about some of those programs I have less familiarity with. I don't see how these tests can be taken seriously.

 

Wow, I just watched the video myself. Outpost was in learning mode the entire time. AV comparatives should know better! I don't know if Outpost will pass the test with it's protection enabled, but it should be given that it certainly is not going to pass any test with it's protection disabled. Learning Mode = most of Outpost's protection disabled. I'm surprised it past some of their test! I don't know if it would have changed the results, but Outpost should have never been tested in Learning Mode for test results that were going to be published on the web. The testing methodology in Outpost's case was flawed. Outpost is not meant to be used in the manner it was used in. I think a product should be tested in the manner it was intended to be used in. Any product can fail in the hands of a negligent operator. I'm not sure what Outpost could change to avoid this. I have not used Outpost in a few years. Like I said, I don't know if it would have changed the results.

 

To quote the Online Armor manual:

 

An anti-executable will not prevent programs that are already allowed to run from piggybacking on other programs that can connect out.

 

But it will prevent them from executing before they can inject themselves into other processes. He also mentions DLL control, which is also helpful. So yes, an anti-executable will in fact help defend against piggybacking.

 

Now doesn't it make you wonder about those other programs your not so familiar with? If they screwed up once that badly, why not on other programs? Makes me start to rethink their AV tests too.