New MRG test results

Discussion in 'other anti-malware software' started by Dark Star 72, Jun 23, 2010.

Thread Status:
Not open for further replies.
  1. Sveta MRG

    Sveta MRG Registered Member

    Joined:
    Aug 16, 2009
    Posts:
    209
    Day 5 results published.

    Regards,
    Sveta
     
  2. ALiasEX

    ALiasEX Registered Member

    Joined:
    Mar 30, 2010
    Posts:
    240
    Browser Security / ID Protection Applications results are displayed twice in the Day 5 .pdf.

    No System / Internet Security Applications results.
     
  3. Dark Star 72

    Dark Star 72 Registered Member

    Joined:
    May 27, 2007
    Posts:
    771
    They on mine, suggest you check if you have day 5 or 4.

    Also read the day 5 notes and notice the new adjusted Zemana reults in both categories :thumb:

    Edit: My bad, I see what you mean. They have half of it correct but have a mix up in the Browser Protection section.
    No doubt Sveta will correct it
     
    Last edited: Jun 29, 2010
  4. Sveta MRG

    Sveta MRG Registered Member

    Joined:
    Aug 16, 2009
    Posts:
    209
    Thanks for the heads up guys:thumb:

    Problem fixed, new report available.

    Regards,
    Sveta
     
  5. cruelsister

    cruelsister Registered Member

    Joined:
    Nov 6, 2007
    Posts:
    1,519
    Location:
    Paris
    I have a question regarding the testing of Keyscrambler. KS by design will never warn about a keylogger being in place, but should instead send scrambled data to whomever is listening.

    Can you please tell me on what basis Keyscrambler was tested?

    Thank you for any help.
     
  6. ALiasEX

    ALiasEX Registered Member

    Joined:
    Mar 30, 2010
    Posts:
    240
    o_O I don't know how I missed that there were indeed System / Internet Security Applications results.

    Anyway, thanks for fixing the error Sveta and again, thanks for these tests.
     
  7. Coolio10

    Coolio10 Registered Member

    Joined:
    Sep 1, 2006
    Posts:
    1,124
    Sveta do you have a screenshot of what alert Kaspersky shows?
     
  8. fax

    fax Registered Member

    Joined:
    May 30, 2005
    Posts:
    3,899
    Location:
    localhost
    In this thread post #42

     
  9. CloneRanger

    CloneRanger Registered Member

    Joined:
    Jan 4, 2006
    Posts:
    4,979
    I notice the vendors ID are not showing lately ?

    mrgp.gif

    Updated info on pass/fail criteria in the PDF :thumb:

    pdf.gif

    Used with previous permission :)
     
  10. Sveta MRG

    Sveta MRG Registered Member

    Joined:
    Aug 16, 2009
    Posts:
    209
    Hi Cruelsister,

    Yes, we understand that KeyScrambler does not alert, but should work silently.

    In terms of your question as to the “basis” of testing KeyScrambler, you could mean two things. Firstly, why did we choose to include the product in the project. Secondly, how did we assess its performance.

    Firstly, we included KeyScrambler as the vendor positions it as a product which can help secure a web browser for online banking etc and prevent user data being captured by criminals. See here for details - http://www.qfxsoftware.com/ks-windows/features.htm

    Secondly, we classify a product as having passed the test if it prevents the data we enter in to the Login fields on the PayPal site (using IE) being captured and then sent to the test page on our website, either silently, or by intercepting the action of the simulator and alerting / prompting the user. If the security application displays alerts, they must be distinguishable from any displayed in response to the control applications.

    Since KeyScrambler does not alert, it just needs to prevent the function of our simulator silently.

    Regards,
    Sveta
     
  11. Baserk

    Baserk Registered Member

    Joined:
    Apr 14, 2008
    Posts:
    1,321
    Location:
    AmstelodamUM
    Hi Sveta,

    Regarding the quoted ^^PDF part;

    'The reason for using this method is to assess whether the security application will, in real use, display alerts that convey enough information to the user to allow them to know what is a malicious application and what is harmless.'

    How exactly have you guys defined 'enough'?

    As it is bluntly stated; "As mentioned earlier, users who have HIPS which are very 'noisy' and alert on too many things, will simply get in the habit of choosing allow, which negates the point of having the application in the first place".

    As this assumption (users who choose a HIPS are not able to use it properly) is presented as a fact, I assume you guys also have a factual idea of what is exactly 'enough information'.
     
  12. fax

    fax Registered Member

    Joined:
    May 30, 2005
    Posts:
    3,899
    Location:
    localhost
    It seems clear in the text that the indicator for enough information is the comparison of information given by the security tool on a legit application and on the malicous one. If they are the same it is not regarded as "enough information". :)

    Pretty straightforward, a pity that they have been established on a "learning by doing" bases instead of been setup before the starting of the testing.
     
  13. Baserk

    Baserk Registered Member

    Joined:
    Apr 14, 2008
    Posts:
    1,321
    Location:
    AmstelodamUM
    So, the mere fact that a HIPS is warning about an 'installation' when you are not 'installing' something, is never deemed enough. :)
     
  14. fax

    fax Registered Member

    Joined:
    May 30, 2005
    Posts:
    3,899
    Location:
    localhost
    As I understood there are two levels:

    - Getting a warning different from the control application;
    - The warning should contain a clear link to the actual thread

    A generic warning about an installation fails the first and the second condition by non qualifying the specific thread link to the installation.

    Usually HIPS will not only warn you of a generic exe launching but also the subsequent activities performed (hooking, driver load, etc). The latter is more significant to assess the ability to clearly inform the user about an actual thread.

    They are not the first that have done this, I think a similar methodology was employed by http://www.pcsecuritylabs.net/ :)

    Cheers,
    Fax
     
  15. Sveta MRG

    Sveta MRG Registered Member

    Joined:
    Aug 16, 2009
    Posts:
    209
    Hi,

    As is explained in the methodology and in posts here, the application must display different alerts for the simulator than it does for the control applications.

    The simulator is malicious in its activity, captures data and then send this out of the test system. The control applications are legitimate, clean, non-malicious applications. It is reasonable to assume an application should be able to distinguish between the two types of activity and alert accordingly.

    If the alerts were the same for the malicious simulator and the harmless control applications, how, in real life, could a user be expected to make a decision as to which to allow or block?

    Regards,
    Sveta
     
  16. andyman35

    andyman35 Registered Member

    Joined:
    Nov 2, 2007
    Posts:
    2,336
    Why must a HIPS display different alerts? As far as the HIPS is concerned both the control app. and the malware tool are the same thing,unsigned,unknown programs that shouldn't be trusted until the user has verified their safety.The fact that one is safe and one isn't is actually irrelevant since they're exactly the same,unknown.The logic of your argument seems to be that there should be more alerts,a policy shown to fail for the average user that just gets pop-up fatigue and allows everything.o_O
     
  17. Sveta MRG

    Sveta MRG Registered Member

    Joined:
    Aug 16, 2009
    Posts:
    209
    We do not look for more alerts / a greater frequency of alerts, instead, we feel having far fewer alerts is more logical. Zemana and Kaspersky manage to do it and pass this test.

    Regards,
    Sveta
     
  18. Sveta MRG

    Sveta MRG Registered Member

    Joined:
    Aug 16, 2009
    Posts:
    209
    Day 6 results published.

    We will be contacting vendors this week to discuss results and ways in which we may be able to help them counter this type of threat once the project has finished.

    Regards,
    Sveta
     
  19. andyman35

    andyman35 Registered Member

    Joined:
    Nov 2, 2007
    Posts:
    2,336
    Will you be making your application generally available at any time for testing purposes?
     
  20. CloneRanger

    CloneRanger Registered Member

    Joined:
    Jan 4, 2006
    Posts:
    4,979
    Vendors ID's are showing up correctly now ;)

    pay.gif

    From the latest PDF results day 6, Zemana is leading the pack :thumb:
     
  21. tobacco

    tobacco Frequent Poster

    Joined:
    Nov 7, 2005
    Posts:
    1,531
    Location:
    British Columbia
    Thanks Sveta

    Kaspersky doing a nice job and Prevx is kickin butt.

    Glad i held off in trying spyshelter :thumbd:
     
  22. Noob

    Noob Registered Member

    Joined:
    Nov 6, 2009
    Posts:
    6,491
    Wow Zemana is KICKING REAL BUTT!! :rolleyes:
    Never tried it before this is changing my mind :D
     
  23. guest

    guest Guest

    @Sveta MRG
    SS developers passed info. They dont want to more test with SS. Is it ethical? Why not removed yet?

    @tobacco
    SS dont need you, you dont need SS. No problem :) SS didnt failed, Tester failed. If you dont know driving, never drive car.

    Maybe they havent got SS's sonars? Zemana has startup protection ability, check attached photo. It asked for corbitek.

    See yourself, with the same 'methodology' I was not able to differ good app from bad the same high risk level for good app.

    I think SpyShelter, Zemana and OA should pass or all 3 shoud fail if that would be fair of course.

    It's only small example but it's not problem to show 1000 similar. Do you want?


    Zemana ask simply; Do not click allow unless this is a legitimate application.

    İs it different than SS? absolutely no.

    Problem is easy and same.
    Result is not important for me, but right methodology is important.



    @Sveta MRG, can you try with same test with digitally signed simulator? It will be interesting for some apps which has good result.
     

    Attached Files:

  24. Sveta MRG

    Sveta MRG Registered Member

    Joined:
    Aug 16, 2009
    Posts:
    209
    Well the tools that we use in our projects will not be available for general public as they can be used (with little modification) for malicious purposes, however we will create a testing tool which will be available for general public. The tool will be more advanced then anything that is currently available and will help users check for themselves if their security application is providing the adequate level of protection that suites individual user's needs.

    The tool should be available for download by the end of next quarter.

    Regards,
    Sveta
     
  25. Noob

    Noob Registered Member

    Joined:
    Nov 6, 2009
    Posts:
    6,491
    Guys, Zemana can only detect new activity after being installed right?
    I remember reading somewhere that if you are infected already it CAN'T detect it :rolleyes:
     
Thread Status:
Not open for further replies.
  1. This site uses cookies to help personalise content, tailor your experience and to keep you logged in if you register.
    By continuing to use this site, you are consenting to our use of cookies.