New MRG test results

Day 5 results published.

Regards,
Sveta

 

Browser Security / ID Protection Applications results are displayed twice in the Day 5 .pdf.

No System / Internet Security Applications results.

 

They on mine, suggest you check if you have day 5 or 4.

Also read the day 5 notes and notice the new adjusted Zemana reults in both categories

Edit: My bad, I see what you mean. They have half of it correct but have a mix up in the Browser Protection section.
No doubt Sveta will correct it

 

Thanks for the heads up guys

Problem fixed, new report available.

Regards,
Sveta

 

I have a question regarding the testing of Keyscrambler. KS by design will never warn about a keylogger being in place, but should instead send scrambled data to whomever is listening.

Can you please tell me on what basis Keyscrambler was tested?

Thank you for any help.

 

I don't know how I missed that there were indeed System / Internet Security Applications results.

Anyway, thanks for fixing the error Sveta and again, thanks for these tests.

 

Sveta do you have a screenshot of what alert Kaspersky shows?

 

In this thread post #42

 

I notice the vendors ID are not showing lately ?



Updated info on pass/fail criteria in the PDF



Used with previous permission

 

Hi Cruelsister,

Yes, we understand that KeyScrambler does not alert, but should work silently.

In terms of your question as to the “basis” of testing KeyScrambler, you could mean two things. Firstly, why did we choose to include the product in the project. Secondly, how did we assess its performance.

Firstly, we included KeyScrambler as the vendor positions it as a product which can help secure a web browser for online banking etc and prevent user data being captured by criminals. See here for details - http://www.qfxsoftware.com/ks-windows/features.htm

Secondly, we classify a product as having passed the test if it prevents the data we enter in to the Login fields on the PayPal site (using IE) being captured and then sent to the test page on our website, either silently, or by intercepting the action of the simulator and alerting / prompting the user. If the security application displays alerts, they must be distinguishable from any displayed in response to the control applications.

Since KeyScrambler does not alert, it just needs to prevent the function of our simulator silently.

Regards,
Sveta

 

Hi Sveta,

Regarding the quoted ^^PDF part;

'The reason for using this method is to assess whether the security application will, in real use, display alerts that convey enough information to the user to allow them to know what is a malicious application and what is harmless.'

How exactly have you guys defined 'enough'?

As it is bluntly stated; "As mentioned earlier, users who have HIPS which are very 'noisy' and alert on too many things, will simply get in the habit of choosing allow, which negates the point of having the application in the first place".

As this assumption (users who choose a HIPS are not able to use it properly) is presented as a fact, I assume you guys also have a factual idea of what is exactly 'enough information'.

 

It seems clear in the text that the indicator for enough information is the comparison of information given by the security tool on a legit application and on the malicous one. If they are the same it is not regarded as "enough information".

Pretty straightforward, a pity that they have been established on a "learning by doing" bases instead of been setup before the starting of the testing.

 

So, the mere fact that a HIPS is warning about an 'installation' when you are not 'installing' something, is never deemed enough.

 

As I understood there are two levels:

- Getting a warning different from the control application;
- The warning should contain a clear link to the actual thread

A generic warning about an installation fails the first and the second condition by non qualifying the specific thread link to the installation.

Usually HIPS will not only warn you of a generic exe launching but also the subsequent activities performed (hooking, driver load, etc). The latter is more significant to assess the ability to clearly inform the user about an actual thread.

They are not the first that have done this, I think a similar methodology was employed by http://www.pcsecuritylabs.net/

Cheers,
Fax

 

Hi,

As is explained in the methodology and in posts here, the application must display different alerts for the simulator than it does for the control applications.

The simulator is malicious in its activity, captures data and then send this out of the test system. The control applications are legitimate, clean, non-malicious applications. It is reasonable to assume an application should be able to distinguish between the two types of activity and alert accordingly.

If the alerts were the same for the malicious simulator and the harmless control applications, how, in real life, could a user be expected to make a decision as to which to allow or block?

Regards,
Sveta

 

Why must a HIPS display different alerts? As far as the HIPS is concerned both the control app. and the malware tool are the same thing,unsigned,unknown programs that shouldn't be trusted until the user has verified their safety.The fact that one is safe and one isn't is actually irrelevant since they're exactly the same,unknown.The logic of your argument seems to be that there should be more alerts,a policy shown to fail for the average user that just gets pop-up fatigue and allows everything.

 

We do not look for more alerts / a greater frequency of alerts, instead, we feel having far fewer alerts is more logical. Zemana and Kaspersky manage to do it and pass this test.

Regards,
Sveta

 

Day 6 results published.

We will be contacting vendors this week to discuss results and ways in which we may be able to help them counter this type of threat once the project has finished.

Regards,
Sveta

 

Will you be making your application generally available at any time for testing purposes?

 

Vendors ID's are showing up correctly now



From the latest PDF results day 6, Zemana is leading the pack

 

Thanks Sveta

Kaspersky doing a nice job and Prevx is kickin butt.

Glad i held off in trying spyshelter

 

Wow Zemana is KICKING REAL BUTT!!
Never tried it before this is changing my mind

 

@Sveta MRG
SS developers passed info. They dont want to more test with SS. Is it ethical? Why not removed yet?

@tobacco
SS dont need you, you dont need SS. No problem SS didnt failed, Tester failed. If you dont know driving, never drive car.

Maybe they havent got SS's sonars? Zemana has startup protection ability, check attached photo. It asked for corbitek.

See yourself, with the same 'methodology' I was not able to differ good app from bad the same high risk level for good app.

I think SpyShelter, Zemana and OA should pass or all 3 shoud fail if that would be fair of course.

It's only small example but it's not problem to show 1000 similar. Do you want?


Zemana ask simply; Do not click allow unless this is a legitimate application.

İs it different than SS? absolutely no.

Problem is easy and same.
Result is not important for me, but right methodology is important.



@Sveta MRG, can you try with same test with digitally signed simulator? It will be interesting for some apps which has good result.

 

Well the tools that we use in our projects will not be available for general public as they can be used (with little modification) for malicious purposes, however we will create a testing tool which will be available for general public. The tool will be more advanced then anything that is currently available and will help users check for themselves if their security application is providing the adequate level of protection that suites individual user's needs.

The tool should be available for download by the end of next quarter.

Regards,
Sveta

 

Guys, Zemana can only detect new activity after being installed right?
I remember reading somewhere that if you are infected already it CAN'T detect it