MRG - new test results

Sveta I agree with you. I once posted "The damage of fanboys advises" in which I critised the near mythical support some applications got at Wilders.

From the applications I mentioned the SBIE and Comodo users responded most fiercely

Expect some critism and emotional reponses (when you not in favour you are against them)

Regards Kees

 

So Sandboxie shouldn't be included because a small number of users may configure it in a way which would prevent FM from executing? Ermm...that is not a reason to exclude it.

 

No you misunderstood my point there.
I'm saying that the test scenario isn't a likely real-world one,where an average user would have SBIE and nothing else on their system.In it's default configuration it certainly won't restrict the malware from running,however the bulk of users that would be using it alone will be techie types that'll configure it for higher security.With restrictions in place it would prevent FM from even executing.

At risk of inciting the wrath of Kees I'd just like it tested in a manner that most mirrors it's likely usage.

 

I understand what you're saying, but by following that logic Bufferzone, DW, Geswall, Prevx, Safecentral, Spyshelter, Trusteer and Trustdefender also shouldn't have been tested. Or is it ok that they were tested?

 

I dont think DW and Geswall should have been tested in this group. I dont know what the others are. But DW and Geswall are HIPS not keyloggers. If they were going to toss HIPS in they should have tested Threatfire and Mamutu as well.

 

DW and Geswall are policy based HIPS, but DW has specific keylogger, clipboard and screengrabber protection. Threatfire and Mamutu are behaviour blockers, not HIPS.

It seems everybody can make a case for why an app should or shouldn't be included.

 

Well those other products make claims regarding the prevention of private information being sent or accessed. Sandboxie is pretty much designed for one thing, especially in default configuration, and that is to protect the system from changes made by programs running in the sandbox. Period.

 

And yet if you use Sandboxie in a particular manner, even in its default configuration it can prevent FM activity. So it can protect from this FM, but unless you specifically use it in a manner to protect you, it doesn't.
We can all keep arguing the toss on this issue, but it doesn't matter. It has been tested, it didn't prevent the FM activity, so now people know...just to enlighten those that didn't previously know that it wouldn't protect them.

 

Ok so next time lets toss in Mamutu and Threatfire in. Even though they are behaviour blockers, they should stop FM activity or anything else that "could" stop it if configured properly.

 

It's true that Sandboxie does not try to detect key or data logging. However, prudent use of Sandboxie involves the "delete sandbox" function, especially when switching from casual browsing to banking.

Your test therefore would seem to represent someone who is using Sandboxie, but not it to its full extent. As such, it is probably not an adequate response to those who praise the robustness of Sandboxie, as they probably take a more rigorous approach to using Sandboxie, and delete their sandboxes often.

 

Bufferzone definitely not,DW and Geswall probably shouldn't IMO.Prevx should since Safeonline is specifically designed to prevent browser hijacking,etc.As for the others I'm not familiar with their workings so couldn't say.

 

Sveta,

Was DefenseWall tested in normal mode or the special anti-keylogger go banking/shopping mode

Regards

Kees

 

DefenseWall HIPS was tested in Normal Mode (default settings).

Regards,
Sveta

 

But not Avast.
Was Avast deliberately excluded or just not included? Just another curious poster......

 

Avast will be included in our next round of testing.

Regards,
Sveta

 

I want to see the more detailed aspects

 

If you don't understand the concept or won't read about how to use Sandboxie then it won't protect against the test and all your test has managed to do is make those that dislike Sandboxie salivate with utter glee.

Yes you contacted Tzuk and he was decent enough to reply:

So you refused to use the simple measure of deleting the sandbox before proceeding with the test.

Maybe Tzuk should bring out a default Sandboxie where only browsers have start/run/internet access just to accommodate testers that have no idea of Sandboxie's capabilities.

 

A few people have contacted me asking "Why did OA fail?"

The simple answer is "I don't know." The tests results are vague.

It would be interesting if MRG were to say why they failed an application at least. This would increase the credibility of the results, as well as let vendors know - is there a useability issue, or a security issue, or both.

OA could have failed for two reasons:

a) OA simply failed, nothing detected, no warning.
b) OA threw up a prompt - which was deemed not informative enough - and it failed.

I'm hoping for case (b).

This sort of testing is a nice idea, but the methodology is a little weak.


Mike


Edit: Just read comments over at the MRG forum...

Seems to answer the question.

 

Do people really care about these tests anymore? They've shown that they are not capable of properly testing applications and that they simply do not understand how the programs they are testing work. They also publish biased tests and change testing methodology during and after tests to suit their biases. I've looked at some of the tests they've performed and honestly cannot help but laugh. This recent test and the handling of Comodo really shows how "professional" they really are.

 

Sorry Peter I don't understand what do you mean by need to be installed? The programs needs to be installed first no way!

TH

 

Yes, I think they do. Most people are not qualified to test applications and have neither the time, nor resources to do so. MRG, Matousec, Creer, Escalader and everyone else who makes good-faith attempts to test software perform a service for the community at large.

Reviewing the methodology, we have first of all the good:

1 - Testing on consistent platform.
2 - Not reusing the platform (install, uninstall, install).
3 - Testing with out of box settings - realistic. We have people who have used OA for some time, for example that didnt know about certain features.


The bad:

1 - There is no detail in the results. An application that completely fails is treated the same way as an application that fails due to a badly worded prompt.

2 - The prompts that may be badly worded introduces an element of subjectivity which is not open to scrutiny or comparison.

3 - The vendor feedback didn't happen in our case at least.

I can understand not wanting to circulate the test app (not sure how Comodo got hold of it, and that should probably be explained to avoid accusations of bias) - however, what this now comes down to is that we have a claim from a party to have tested a product, and gotten a result where there is no possible way to analyse or verify it in any way at all.

This could be fixed quite simply by providing more details of prompts that were, or were not issued by the security application.

I haven't dealt much with the MRG guys so I can't comment on arguments you raise about bias, but in my limited dealings with them on prior tests they seemed to be ok guys.

As for how they handle Comodo: Having previously had a few run-ins with Comodo myself I would say ignoring them is a pretty sound strategy.

I am curious however, why Comodo results were pulled after a disgreement with them on details that nobody else seems to have.


Mike

 

I think he's being facetious, TH.

 

So was I. Can't help but laugh!

TH

 

It was a satirical statement.

Inference (in my opinion) was: If a tester is too inept to use security software properly, he is probably too inept to even INSTALL it properly. I heartily agree.

IMO, this tester lacks the technical expertise needed to properly test these kinds of security apps.