Bad news on Online Armor

[MikeNash]

Hi Everyone,

Online Armor did not do as well in the Leaktests over at Matousec as we thought. Please, see this post over at the Online Armor forums for more details.

http://support.online-armor.com/forums/viewtopic.php?p=11800#11800

Any problems, questions - please let me know.


Mike

[hiro]

Thank you Mike, this is honest from you, this is example how privacy protect software developer must communicate with user to construct trust in this so delicate area (how to protect online security and privacy).
It doesn't matter if software is FREE or pay (or first or last on leaktest stairs)!
Thanks again!

[GES/POR]

I disagree. There is such a thingh as 2 much honesty. Every software is bound 2 have bugs and Talemu is the most dedicated company 2 resolve any type of issue lightening fast. This update didn't need 2 be brought 2 our attention this loud. However it is interresting 2 know the best can be inproved and therefore my thanks 2 keep us posted.

[Thanasis159]

Thank you very much for your honesty... II really appreciate it! Keep up the good work!

[Peter2150]

I have two reactions:

1) I applaud Mike in his candor to post problems as well as triumphs.

2) It really makes me question the value of the Matousec tests. True there can be debate about protected usermode hooks vs kernel mode hooks, but the tests are tests, and eitherr the software passed or not. If Matousec is saying a product isn't as good using usermode hooks, then the tests should reflect that weakness. But to run the tests, have a program pass them, and the say oops, the results aren't that good because..... is just plain bogus testing IMHO.

To trust the tests you have to trust the tester. NicM is a good example of a good tester. I have to question Matousec's tests.

Pete

[Dwarden]

well at least he found out his mistake and most likely works on correcting these results and updating test methology ...

[MikeNash]

{QUOTE-> well at least he found out his mistake and most likely works on correcting these results and updating test methology ... <-QUOTE}

To be fair, They tested two versions of Online Armor - first of all we came in the middle of the pack. Then, we gave an updated version where had expanded our protection, but also we had added some code to protect our remaining usermode hooks.

We did not advise Matousec that we had added this protective code - probably, we should have done so.


Mike

[Perman]

Hi, folks: We, users will deliver the ultimate verdict, not the testers. But any software needs an mirror such as these tests to see how they look, and to make necessary improvements. In OA, trusts remain. :thumb: :thumb:

[Bio-Hazard]

Hello!

I think is show lot of respect from Mike and Tallemu towards their users to publicly say these and not trying to hide behind any explenations. :thumb: I dont know much about testing, but i have to agree with Peter that something went wrong and it should have been picked earlier. Online-Armor still is the corner stone of my security set-up. Now we just have to wait for the updated results and see what happens. Atleast now it doesnt come as a suprise to people when it happens.

Kristian

[danny9]

I'm still trialing Online Armor and I appreciate the honesty Mike has shown.
To me it's refreshing to see this.
Thanks, Dan ;)

[Graystoke]

I hope this isn't too far off topic. Would you consider Online Armor's firewall to be among the top tier, or middle of the pack of firewalls? I don't pay too much attention to Mr. Matousec's tests, so I'm not concerned with that. I ran the 30 day trial and liked it, but not sure I want to spend around $40 since I'm using a pretty good free firewall now.

[trjam]

OA may be one of the best products created, until, Mike added Kaspersky and the only other thing is the modules or tabs that ask for scanning specific tasks like email. The firewall is the best and I wish that is what he concentrated on and had a Vista version ready. I would buy it in a second then.

[Peter2150]

{QUOTE-> I hope this isn't too far off topic. Would you consider Online Armor's firewall to be among the top tier, or middle of the pack of firewalls? I don't pay too much attention to Mr. Matousec's tests, so I'm not concerned with that. I ran the 30 day trial and liked it, but not sure I want to spend around $40 since I'm using a pretty good free firewall now. <-QUOTE}

Yes, for me it is. I also don't care about Matousec's test. It does it job as do most firewalls, but the traffic led, and firewall status display are simply unrivaled. Check it out.

[Peter2150]

{QUOTE-> OA may be one of the best products created, until, Mike added Kaspersky and the only other thing is the modules or tabs that ask for scanning specific tasks like email. The firewall is the best and I wish that is what he concentrated on and had a Vista version ready. I would buy it in a second then. <-QUOTE}

Not sure why mention Kaspersky. If you are thinking about the great Iswift controversy, OA doesn't use it. One can license the engine, and additionally other modules such as Iswift, but OA only uses the engine.

Pete

[MikeNash]

{QUOTE-> OA may be one of the best products created, until, Mike added Kaspersky and the only other thing is the modules or tabs that ask for scanning specific tasks like email. The firewall is the best and I wish that is what he concentrated on and had a Vista version ready. I would buy it in a second then. <-QUOTE}

Hi trjam,

Can you be more specific on what you dont like regarding the modules/tabs (PM me, start a thread at our forums, etc) - I'm not sure I follow but I am glad you like the firewall :)

You know you can turn off mail filtering, and you dont need to buy the AV version?

Mike

[trjam]

I think I have my answers from your forum and will be patiently awaiting a Vista version. You are a very dedicated and honest vendor and your product shows it.

[LUSHER]

Well I guess it falls to me to point out the elephant in the room.

Even if Mike didn't disclose this, there was zero chance this thread wouldn't be created anyway , so it was smart to head it off first by posting this. This is something that has to be gotten out of the way.

As for the other comments about not trusting Matousec tests, I never understood where this came from. Leaving aside the whole business practice thing of selling their findings (which you may disagree), it's pretty obvious they are knowledgable and their testing standards is as high as anyone can wish for because they know they will be subjected to intense scrunity given that they are playing with the big boys.

Moreover despite aspersations casted on their age (some are college??), they are mature unlike say certain anti-rootkit authors I shall not name, who engage in flamefests or pointless boasting.

Of course they are not perfect, but I would trust their testing because clearly they are a professional outfit or at least aspire to those standards.
Something I can't say for many of other tests as useful as they may be.

Anyway I agree leak testing isn't everything, and some might not realize this with all the focus on leak testing (cos it is easiest to do and understand) but Matousec agrees as well. Comodo might have the highest leak test score, but the firewall with the best overall score in the Windows Personal Firewall Analysis project is not Comodo but ZoneAlarm PRO!

[MikeNash]

Revised results are now in at Matousec Security.

http://www.matousec.com/projects/windows-personal-firewall-analysis/leak-tests-results.php

Online Armor is now in 7th place on the table, with a rating of "Very Good".

Mike

[innerpeace]

Thanks for the update Mike. That is still very respectable for a brand spanking new firewall.

[Graystoke]

{QUOTE-> Yes, for me it is. I also don't care about Matousec's test. It does it job as do most firewalls, but the traffic led, and firewall status display are simply unrivaled. Check it out. <-QUOTE}


Thanks Peter2150. :)

[GES/POR]

{QUOTE-> Revised results are now in at Matousec Security.

http://www.matousec.com/projects/windows-personal-firewall-analysis/leak-tests-results.php

Online Armor is now in 7th place on the table, with a rating of "Very Good".

Mike <-QUOTE}

Dang! I didn't c that coming. :'(

[MikeNash]

{QUOTE-> Dang! I didn't c that coming. :'( <-QUOTE}

Yes, unfortunately our firewall is merely "Very Good" at leaktest prevention, and not "Excellent" :)

We've already started to fix it, and indeed a new version will be uploaded to the beta team (and released) which will gain us back some valuable places.

[wat0114]

{QUOTE-> Dang! I didn't c that coming. :'( <-QUOTE}

What is there to be concerned about? Leaktest results are not the only criteria that should be considered when choosing a product. You need to be comfortable using it, it needs to perform its intended task very well, it should run stable on your machine and the vendor should offer good support on it. it would seem OA falls very nicely into all of these categories.

[Seer]

I just saw the thread title and expected some real "bad news on OA", but this is just ridiculous. :P 7th place on Matousec list?! LMAO. Funny how this guy (David M) became the one and only beacon in a world of firewalls/leaking/plumbing so quickly ::) Our dear vendors are sooo afraid of his verdicts... this guy alone literally shapes the firewall market. It's all, of course, users' fault.

BTW, 7th place is quite excellent IMO... more than enough for ALL of us. I know, the 3rd place would make Tall Emu's sales department a bit busier... ;)

Relax, Mike & Tall Emu. ;) I'm not sure if this topic is even worth a thread...

[GES/POR]

{QUOTE-> What is there to be concerned about? Leaktest results are not the only criteria that should be considered when choosing a product. You need to be comfortable using it, it needs to perform its intended task very well, it should run stable on your machine and the vendor should offer good support on it. it would seem OA falls very nicely into all of these categories. <-QUOTE}

No concern at all but i think leaking is a serious issue for firewalls since malware wich is not detected by tons of scanners needs to be stopped from communicating from a personal pc to another wich happens rather often on most pc's. If im not mistaking the primary function of HIPS is total control of one's pc meaning not leaking as well.

For Matousec i wish he made some kind of table of other aspects of security that is tested such as termination,etc.

Funny 2 c everytime a fav product fails some tests then the tester get's bashed while infact he is only trying to improve security for all. Y not bash those leaktest authors or malware writers?

Still i am content with OA's results and have all faith theyll climb the ladder soon enough.

[GES/POR]

{QUOTE-> Hi trjam,

Can you be more specific on what you dont like regarding the modules/tabs (PM me, start a thread at our forums, etc) - I'm not sure I follow but I am glad you like the firewall :)

You know you can turn off mail filtering, and you dont need to buy the AV version?

Mike <-QUOTE}

It would be sweet to have a another OA AV option such as webwasher or avira. Would this be more 2 ur liking Trjam?

[wat0114]

{QUOTE->
Funny 2 c everytime a fav product fails some tests then the tester get's bashed while infact he is only trying to improve security for all. <-QUOTE}

It seems to work both ways. Either the tester gets bashed or the vendor of the tested product is bashed for not providing a secure enough product. I have expressed skepticism towards Matousec's testing, but in the end I would agree with you that he is only trying to help. The problem I see is that some people get so caught up in the apparent importance of leaktesting that they can lose sight of other, important factors, regarding proper security practices.

[screamer]

{QUOTE-> Well I guess it falls to me to point out the elephant in the room.

Even if Mike didn't disclose this, there was zero chance this thread wouldn't be created anyway , so it was smart to head it off first by posting this. This is something that has to be gotten out of the way.

<-QUOTE}

While I don't use OA, I do read many of Mike Nash's posts. I've come to regard him as a true gentlemen in the security business. He appears to be dedicated to his app and his customer base. You really can't ask for more than this.

...screamer

[rdsu]

MikeNash,

Thanks for being a honest person, and keep your very good work... ;)

:thumb:

[MikeNash]

{QUOTE-> I just saw the thread title and expected some real "bad news on OA", but this is just ridiculous. :P 7th place on Matousec list?! LMAO. Funny how this guy (David M) became the one and only beacon in a world of firewalls/leaking/plumbing so quickly ::) Our dear vendors are sooo afraid of his verdicts... this guy alone literally shapes the firewall market. It's all, of course, users' fault.

BTW, 7th place is quite excellent IMO... more than enough for ALL of us. I know, the 3rd place would make Tall Emu's sales department a bit busier... ;)

Relax, Mike & Tall Emu. ;) I'm not sure if this topic is even worth a thread... <-QUOTE}

Hi Seer,

I agree that this is just one test, and that 7th place is not so bad. However, we *were* in third place and I had published that fact, so I had to correct it when the real tests come out.

I'm not overly paniced about these results - but since I'd been saying "Yay! Woo! We're in third place"... even on our website - had to change it.


Mike

[xStylezx]

I cant wait to give this a try when it comes out for vista.Been checking the site and forums waiting,so hopefully soon.

[dja2k]

Well all I have to say is that Mike has always been honest. Secondly, Online Armor can only get better from here! :D

dja2k

[Stem]

Personally, I am not very interested in leak-test results (or the capabilities of the firewall in this area).

My main concern with OA firewall as always been the packet filtering capability (or lack off). As an SPI yet been added?, are even any sort of packet filter to at least filter out illigal/malformed packets?

[Doc Serenity]

Stem makes a good point.
But what products do already have this ability?

[Stem]

{QUOTE-> But what products do already have this ability? <-QUOTE}Most firewalls have this ability, with an SPI and/or filters for bad packets, such as comodo / Jetico / ZA / outpost /etc etc. They can, and do vary on capability (and correct working).

[lucas1985]

{QUOTE-> They can, and do vary on capability (and correct working). <-QUOTE}
Hi Stem,
Have you done some tests on the "grey" area of SPI filtering?

[Doc Serenity]

Thanks for your reply.
And Lucas brings up a good question.
This would be interesting reading.
Thanks.
Doc

[Stem]

Hello lucas1985, Doc,
{QUOTE-> Hi Stem,
Have you done some tests on the "grey" area of SPI filtering? <-QUOTE}Yes, I do quite a lot of testing in this area.
I do have a problem with posting results from this, unlike we have now for leak-tests, the methods used for SPI (possible) bypass are not well/ or even documented for users to try, and posting some of the methods I use would probably be against forum TOS, so results could not be easily confirmed by other members.

[MikeNash]

{QUOTE-> Personally, I am not very interested in leak-test results (or the capabilities of the firewall in this area).

My main concern with OA firewall as always been the packet filtering capability (or lack off). As an SPI yet been added?, are even any sort of packet filter to at least filter out illigal/malformed packets? <-QUOTE}

Hi Stem

I am not 100% sure, but I do not believe this currently exists in OA's firewall.

Mike

[appster]

I'm looking for a 'lighter' FW than ZAP 7 which clearly slows down my laptop (WinXP, Pent M 1.8, 512MB RAM), which of Matousec's Exc or VG rated FWs would you guys recommend? :-\

[Seer]

Hello Mike. :)

{QUOTE-> I'm not overly paniced about these results - but since I'd been saying "Yay! Woo! We're in third place"... even on our website - had to change it. <-QUOTE}

So you're really after those rankings at Matousec then? Good to know you're serious about outbound protection. :thumb:
The thing is you see, I have a folder on my machine with different installations organized in subfolders by type of app - Firewalls, AVs, HIPS, etc. Now, I have OA trial installation placed under 'Firewalls'. Should I maybe consider moving it to 'HIPS'?
I have always considered OA to be a firewall so I would rather like to see it moving more in that direction. So generally, I have to agree with Stem on packet filtering, although I am a little surprised by his question regarding SPI. I am not very familiar with OA rules and protocol handling, but I'm just assuming at least SPI for TCP is present... ??? Well, I would actually have to install OA in order to continue on this thread...

See ya, :)

[Stem]

{QUOTE-> ......although I am a little surprised by his question regarding SPI. I am not very familiar with OA rules and protocol handling, but I'm just assuming at least SPI for TCP is present... ??? <-QUOTE}On my last installation of OA,.. OA was using a "Pseudo SPI", this is basically a table of IP`s that have been connected to, and inbound packets are allowed based on this table. This in itself will block unsolicited inbound from IP`s not connect to (if no open inbound allow rule is in place), but will not filter for bad/illigal/spoofed packets.

[Seer]

Got it, Stem. So that's how "pseudo-SPI" works. It only scans for packet specifications (port, IP) on a whitelist principle instead of the actual contents of a packet. This is in fact, one half of the full SPI. ;D Now I see why full SPI cannot be implemented for conectionless protocols.
But enough with the ot... "bad news" is the topic here. :)

Cheers,

[lucas1985]

{QUOTE-> Hello lucas1985, Doc,
Yes, I do quite a lot of testing in this area.
I do have a problem with posting results from this, unlike we have now for leak-tests, the methods used for SPI (possible) bypass are not well/ or even documented for users to try, and posting some of the methods I use would probably be against forum TOS, so results could not be easily confirmed by other members. <-QUOTE}
Well, you could post some general guidelines (within the TOS) and/or general information about the current state of Windows firewalls.