MRG - new test results

Discussion in 'other anti-malware software' started by Dark Star 72, Apr 27, 2010.

Thread Status:
Not open for further replies.
  1. Kees1958

    Kees1958 Registered Member

    Joined:
    Jul 8, 2006
    Posts:
    5,857
    Sveta I agree with you. I once posted "The damage of fanboys advises" in which I critised the near mythical support some applications got at Wilders.

    From the applications I mentioned the SBIE and Comodo users responded most fiercely

    Expect some critism and emotional reponses (when you not in favour you are against them) :D

    Regards Kees
     
  2. Scoobs72

    Scoobs72 Registered Member

    Joined:
    Jul 16, 2007
    Posts:
    1,113
    Location:
    Sofa (left side)
    So Sandboxie shouldn't be included because a small number of users may configure it in a way which would prevent FM from executing? Ermm...that is not a reason to exclude it.
     
  3. andyman35

    andyman35 Registered Member

    Joined:
    Nov 2, 2007
    Posts:
    2,336
    No you misunderstood my point there.
    I'm saying that the test scenario isn't a likely real-world one,where an average user would have SBIE and nothing else on their system.In it's default configuration it certainly won't restrict the malware from running,however the bulk of users that would be using it alone will be techie types that'll configure it for higher security.With restrictions in place it would prevent FM from even executing.

    At risk of inciting the wrath of Kees I'd just like it tested in a manner that most mirrors it's likely usage.
     
  4. Scoobs72

    Scoobs72 Registered Member

    Joined:
    Jul 16, 2007
    Posts:
    1,113
    Location:
    Sofa (left side)
    I understand what you're saying, but by following that logic Bufferzone, DW, Geswall, Prevx, Safecentral, Spyshelter, Trusteer and Trustdefender also shouldn't have been tested. Or is it ok that they were tested?
     
  5. whitedragon551

    whitedragon551 Registered Member

    Joined:
    Sep 30, 2008
    Posts:
    3,264
    Location:
    USA
    I dont think DW and Geswall should have been tested in this group. I dont know what the others are. But DW and Geswall are HIPS not keyloggers. If they were going to toss HIPS in they should have tested Threatfire and Mamutu as well.
     
  6. Scoobs72

    Scoobs72 Registered Member

    Joined:
    Jul 16, 2007
    Posts:
    1,113
    Location:
    Sofa (left side)
    DW and Geswall are policy based HIPS, but DW has specific keylogger, clipboard and screengrabber protection. Threatfire and Mamutu are behaviour blockers, not HIPS.

    It seems everybody can make a case for why an app should or shouldn't be included.
     
  7. mvario

    mvario Registered Member

    Joined:
    Sep 16, 2008
    Posts:
    339
    Location:
    Haddonfield, IL
    Well those other products make claims regarding the prevention of private information being sent or accessed. Sandboxie is pretty much designed for one thing, especially in default configuration, and that is to protect the system from changes made by programs running in the sandbox. Period.
     
  8. Scoobs72

    Scoobs72 Registered Member

    Joined:
    Jul 16, 2007
    Posts:
    1,113
    Location:
    Sofa (left side)
    And yet if you use Sandboxie in a particular manner, even in its default configuration it can prevent FM activity. So it can protect from this FM, but unless you specifically use it in a manner to protect you, it doesn't.
    We can all keep arguing the toss on this issue, but it doesn't matter. It has been tested, it didn't prevent the FM activity, so now people know...just to enlighten those that didn't previously know that it wouldn't protect them.
     
  9. whitedragon551

    whitedragon551 Registered Member

    Joined:
    Sep 30, 2008
    Posts:
    3,264
    Location:
    USA
    Ok so next time lets toss in Mamutu and Threatfire in. Even though they are behaviour blockers, they should stop FM activity or anything else that "could" stop it if configured properly.
     
  10. tzuk

    tzuk Developer

    Joined:
    Jul 4, 2004
    Posts:
    34
    It's true that Sandboxie does not try to detect key or data logging. However, prudent use of Sandboxie involves the "delete sandbox" function, especially when switching from casual browsing to banking.

    Your test therefore would seem to represent someone who is using Sandboxie, but not it to its full extent. As such, it is probably not an adequate response to those who praise the robustness of Sandboxie, as they probably take a more rigorous approach to using Sandboxie, and delete their sandboxes often.
     
  11. andyman35

    andyman35 Registered Member

    Joined:
    Nov 2, 2007
    Posts:
    2,336
    Bufferzone definitely not,DW and Geswall probably shouldn't IMO.Prevx should since Safeonline is specifically designed to prevent browser hijacking,etc.As for the others I'm not familiar with their workings so couldn't say.
     
    Last edited: Apr 28, 2010
  12. Kees1958

    Kees1958 Registered Member

    Joined:
    Jul 8, 2006
    Posts:
    5,857
    Sveta,

    Was DefenseWall tested in normal mode or the special anti-keylogger go banking/shopping mode

    Regards

    Kees
     
  13. Sveta MRG

    Sveta MRG Registered Member

    Joined:
    Aug 16, 2009
    Posts:
    209

    DefenseWall HIPS was tested in Normal Mode (default settings).

    Regards,
    Sveta
     
  14. jwcca

    jwcca Registered Member

    Joined:
    Dec 6, 2003
    Posts:
    772
    Location:
    Toronto
    But not Avast.
    Was Avast deliberately excluded or just not included? Just another curious poster......o_O
     
  15. Sveta MRG

    Sveta MRG Registered Member

    Joined:
    Aug 16, 2009
    Posts:
    209
    Avast will be included in our next round of testing.

    Regards,
    Sveta
     
  16. Noob

    Noob Registered Member

    Joined:
    Nov 6, 2009
    Posts:
    6,490
    I want to see the more detailed aspects :D
     
  17. Franklin

    Franklin Registered Member

    Joined:
    May 12, 2005
    Posts:
    2,517
    Location:
    West Aussie
    If you don't understand the concept or won't read about how to use Sandboxie then it won't protect against the test and all your test has managed to do is make those that dislike Sandboxie salivate with utter glee.

    Yes you contacted Tzuk and he was decent enough to reply:
    So you refused to use the simple measure of deleting the sandbox before proceeding with the test.

    Maybe Tzuk should bring out a default Sandboxie where only browsers have start/run/internet access just to accommodate testers that have no idea of Sandboxie's capabilities.
     
  18. MikeNash

    MikeNash Security Expert

    Joined:
    Jun 9, 2005
    Posts:
    1,656
    Location:
    Sydney, Australia
    A few people have contacted me asking "Why did OA fail?"

    The simple answer is "I don't know." The tests results are vague.

    It would be interesting if MRG were to say why they failed an application at least. This would increase the credibility of the results, as well as let vendors know - is there a useability issue, or a security issue, or both.

    OA could have failed for two reasons:

    a) OA simply failed, nothing detected, no warning.
    b) OA threw up a prompt - which was deemed not informative enough - and it failed.

    I'm hoping for case (b).

    This sort of testing is a nice idea, but the methodology is a little weak.


    Mike


    Edit: Just read comments over at the MRG forum...

    Seems to answer the question.
     
  19. codylucas16

    codylucas16 Registered Member

    Joined:
    Nov 17, 2009
    Posts:
    267
    Do people really care about these tests anymore? They've shown that they are not capable of properly testing applications and that they simply do not understand how the programs they are testing work. They also publish biased tests and change testing methodology during and after tests to suit their biases. I've looked at some of the tests they've performed and honestly cannot help but laugh. This recent test and the handling of Comodo really shows how "professional" they really are.
     
  20. Peter2150

    Peter2150 Global Moderator

    Joined:
    Sep 20, 2003
    Posts:
    20,534
    A few years back with the earlier concept of Prevx they found that 50% of the users failed to do the correct thing with pop ups. Now we find testers doing the same thing.

    I am waiting for one of the test groups to fail a program because it failed to warn the user it needed to be installed.
     
  21. Triple Helix

    Triple Helix Specialist

    Joined:
    Nov 20, 2004
    Posts:
    12,756
    Location:
    Ontario, Canada
    Sorry Peter I don't understand what do you mean by need to be installed? The programs needs to be installed first no way! :gack: o_O

    TH
     
    Last edited: Apr 29, 2010
  22. MikeNash

    MikeNash Security Expert

    Joined:
    Jun 9, 2005
    Posts:
    1,656
    Location:
    Sydney, Australia
    Yes, I think they do. Most people are not qualified to test applications and have neither the time, nor resources to do so. MRG, Matousec, Creer, Escalader and everyone else who makes good-faith attempts to test software perform a service for the community at large.

    Reviewing the methodology, we have first of all the good:

    1 - Testing on consistent platform.
    2 - Not reusing the platform (install, uninstall, install).
    3 - Testing with out of box settings - realistic. We have people who have used OA for some time, for example that didnt know about certain features.


    The bad:

    1 - There is no detail in the results. An application that completely fails is treated the same way as an application that fails due to a badly worded prompt.

    2 - The prompts that may be badly worded introduces an element of subjectivity which is not open to scrutiny or comparison.

    3 - The vendor feedback didn't happen in our case at least.

    I can understand not wanting to circulate the test app (not sure how Comodo got hold of it, and that should probably be explained to avoid accusations of bias) - however, what this now comes down to is that we have a claim from a party to have tested a product, and gotten a result where there is no possible way to analyse or verify it in any way at all.

    This could be fixed quite simply by providing more details of prompts that were, or were not issued by the security application.

    I haven't dealt much with the MRG guys so I can't comment on arguments you raise about bias, but in my limited dealings with them on prior tests they seemed to be ok guys.

    As for how they handle Comodo: Having previously had a few run-ins with Comodo myself I would say ignoring them is a pretty sound strategy.

    I am curious however, why Comodo results were pulled after a disgreement with them on details that nobody else seems to have.


    Mike
     
  23. Page42

    Page42 Registered Member

    Joined:
    Jun 18, 2007
    Posts:
    6,267
    Location:
    USA
    I think he's being facetious, TH. ;)
     
  24. Triple Helix

    Triple Helix Specialist

    Joined:
    Nov 20, 2004
    Posts:
    12,756
    Location:
    Ontario, Canada
    So was I. :D Can't help but laugh!

    TH
     
  25. bellgamin

    bellgamin Very Frequent Poster

    Joined:
    Aug 1, 2002
    Posts:
    6,673
    Location:
    Hawaii
    It was a satirical statement.

    Inference (in my opinion) was: If a tester is too inept to use security software properly, he is probably too inept to even INSTALL it properly. I heartily agree.

    IMO, this tester lacks the technical expertise needed to properly test these kinds of security apps.
     
Loading...
Thread Status:
Not open for further replies.
  1. This site uses cookies to help personalise content, tailor your experience and to keep you logged in if you register.
    By continuing to use this site, you are consenting to our use of cookies.