Are security software developers lazy to test their own software?

Ok, I have a feeling since long and I think long time lurkers here will agree with me.

Every now n then a new security software will launch with great pump n show, and just after few days we will find people posting on the forums about malware samples bypassing that software. I agree that a bypass here n there doesn,t mean that a software is not decent in protection but still I get surprised when commonly discusssed/ notorious malware/ POCs bypass a newly launchced HIPS/ Sandbox.

I just wonder why the developers don,t bother to test their software against a good no of well known/ tricky malware samples/ POCs before releasing their work to the public. It will give them more credibility indeed.

What do you think?

 

More like some companies have less resources than others but they can use their great community in their advantage

 

I don't think they are lazy. I think that many would see it as fake. How many times have we seen norton paying for and sponsing a test. Whats the reaction around here? "Oh it must be fixed because they payed for it". I think that they are trying to avoid this. Companies like norton have enough money and resources to pay for a test like that. Most people want to see an independant test rather than one from the same company that makes the product.

 

I don't think aigle meant public tests but rather vendors testing their own product's capabilities against nasties and if needed then improving in that particular area.However I do think that they do test their software against malware, maybe not too much against real malware but maybe against ways malware infects a system and also it is very difficult to get hands on each and every piece of malware and testing them against their products.Nevertheless, aigle has a very interesting question.

 

[2¢]There are some vendors who don't even get to the launch and bypass phase.
They just get stuck in the "pump n show" phase.
We all know who they are... promise, pump, pump, promise...
but no new release.
Sometimes I think these vendors are worse than the ones aigle is talking about.[/2¢]

 

Malware is ever evolving. Its like a bad strain of super bug. Its constantly changing. Anti-biotics are unable to keep up with them. Software companies are very much the same. Once they find a new malware it takes time for them to react to it. I'm not a software writer but I would assume that its not as easy as a few keystrokes to combat the newest kernel hook, script or exploit. I think that they do test it against the newest threats but again that changes day to day. How many AV analyst do they have compared to malware writers that probably outnumber them.

 

They've got to focus on selling their software 1st...marketing, convincing end-users that using XXX software would solve 90-99% of their malware problems (some go as far as saying 100%), fund/payment transfer, coupon codes, licensing issues, etc etc

Testing their security software would require time, resources, etc etc. If they had known of every single malware problem existent, that would be "peanuts" but the harsh truth is that no software developer knows each and everything...each having their own field of expertise and as such, mostly focus on what they're good at/comfortable with. What may be considered as 'old-age issues' by some folks here may be new 'surprises' for them...even if they don't admit it upfront all times. Not to mention that they have to find the 'sweet spot' of usability/convenience vs security....they've got to make some sort of trade-off when configuring the 'default setting' for users. You can see this problem within Microsoft software itself - the 'dreaded' UAC for example.

Then again - think about this: If the majority of the users are convinced that 'it works perfect' without knowing the possible bypasses, (how many actually bother?), is it permissible for me to argue that there's not much gain or incentive to pursue the matter with the exception of a minority group of advanced users, techies, geeks and forum board members like the ones lurking and posting here

Furthermore, it does add a spark of healthy discussion (and constant flows of fanboyism) every now and then to hear that their security software has been bypassed by something some folks consider 'trivial'....indirectly posing as a free publicity (negative publicity can be used to an advantage - it may be considered better than having no/little publicity) To add to that, it does help to portray a good image of the company whenever they 'fix' the issue reported by concerned user(s)...it shows that the team/company 'listens to their customers'....more so when the 'fix' comes withing a short period of time.

All-in-all, I think it's fair to call them 'lazy' but who isn't? I think human by nature isn't hardworking unless forced to by requirements or circumstances.

 

LOL, i think i didn't understand the question the first time

 

Yes, I mean exactly same.

Thanks

 

I agree. It's quite upsetting that, despite antimalware apps can detect xyz threat, they fail to successfully block it.

 

There's much more to testing security apps than throwing malware samples at it. An application that works on one OS and software combo can have problems on another. As far as the malware itself is concerned, by the time you count variants, you're somewhere around a half million samples. It's physically impossible for anyone to test just one of each type and have any hope of current results.

Regarding POCs bypassing HIPS, breaking out of sandboxes etc, it's physically impossible to defend every conceivable way malware can defeat some part of the OS and installed apps. Even if you cover every known method, a new one is found, you patch it, then repeat the cycle. Until operating systems are no longer default-permit by design, this will continually repeat. As long as testing is based on the concept of launching the malware, then attempting to contain every possible malicious action, there will always be failures. The real fault here isn't the security apps. It's the default-permit security policy they're trying to enforce. It doesn't matter how well the security apps are designed or how well they're tested. If malicious code is allowed to execute, there are no guarantees.

 

I don't mean to test the software against hundreds of samples.
If i am a developer, i wil move around many forums to look for unique malware samples being reported to bypass other security software, wil grab them and test them against my own software. Needs a bit of time though.

 

The vendors I know a bit more about, are testing there software in several stages. Internal testings (sometimes a two-phase testing), then possibily a public beta testing and then POC by the QA dept before release to the public. Mostly these tests are related to the software itself only. (including compatibility with other apps.)
This is also not very easy due to the several setups users have.
Testing against all possible malware seams very difficult to me if not impossible. So far I have never heard of a software solution detecting everything.

Gerard

 

Security software developers test (internal/beta testing) their solutions, but cannot do much due to various reasons especially (novice) user convenience. Take examples of stolen certificates & zero day vulnerabilities by Stuxnet, signed TDL4 rootkit driver, Patch guard restrictions on x64, etc. Should they protect you against Windows vulnerability as well as bypass patch guard just like TDL3? They can do thing within limits to protect you against every threat (phishing, malwares, MITM, probing, exploits, DDOS, etc.) They have to compromise on possible security vs. usability. If they make tough application, user will complain that it is not user friendly and too much talkative. I do not know which button to press…

You are exposing their whole infrastructure