Small scan test of Hitman pro and Prevx

Discussion in 'other anti-malware software' started by ako, Aug 27, 2010.

Thread Status:
Not open for further replies.
  1. ako

    ako Registered Member

    Joined:
    Nov 16, 2006
    Posts:
    627
    I got recently from http://malwareresearchgroup.com/ a set of 1100 new pieces of malware. (They give sample sets when requested for us at Gizmo's Freeware for testing purposes.)

    Hitman pro found 1040/1100.

    Typically 2-3 AV:s recognized each beast. If we ASSUME that all engines are approximately equal, the scanning performance of each single engine is thus 44%.

    I scanned also with Prevx alone. It found 40%.

    The power of using several scanners is seen. :thumb:
     
  2. clocks

    clocks Registered Member

    Joined:
    Aug 25, 2007
    Posts:
    2,559
    Interesting. I am kind of surprised Prevx did not do better. Hitman Pro is incredible.
     
  3. trjam

    trjam Registered Member

    Joined:
    Aug 18, 2006
    Posts:
    9,057
    Location:
    North Carolina
    what were your settings on Prevx, thanks
     
  4. wearetheborg

    wearetheborg Registered Member

    Joined:
    Nov 14, 2009
    Posts:
    667
    That is a wild assumption that I have massive difficulty believing.

    Malware detection should be very closely correlated amongst the various AV engines.

    I am also skeptical that Prevx would only detect 40% .... maybe some settings tweaks are required.

    EDIT: I take back my comments, I did not realise it was 0-day :p
     
    Last edited: Aug 27, 2010
  5. ako

    ako Registered Member

    Joined:
    Nov 16, 2006
    Posts:
    627
    Medium - low - low
     
  6. ako

    ako Registered Member

    Joined:
    Nov 16, 2006
    Posts:
    627
    It was scanner test, not on-execution test.
     
  7. ako

    ako Registered Member

    Joined:
    Nov 16, 2006
    Posts:
    627
    This is a 0-day set. We would not get 94.5% detection rate if correlation would be high.
     
  8. LagerX

    LagerX Registered Member

    Joined:
    Apr 16, 2008
    Posts:
    540
    I think Prevx will jump to 90+ % when you try to execute those malicious files.

    Btw, I don't take tests seriously? Why?
    When you encounter any malware (one day or another) and your AV/Anti-Malware program detects it, you're good!

    If your protection software has 99.99% detection rate and you fail that one malware, you're infected! That is what matters.
     
  9. Triple Helix

    Triple Helix Webroot Product Advisor

    Joined:
    Nov 20, 2004
    Posts:
    12,011
    Location:
    Ontario, Canada
    Also Prevx strengths is shown when you execute the files instead of just scanning the files! But I don't recommend doing so unless you have an Isolated VM!

    TH

    Edit: LagerX beat me to it ;)
     
  10. Matthijs5nl

    Matthijs5nl Guest

    Why would you be skeptical. Prevx is a great program and I absolutely love it. But 40% is not extremely bad in my opinion. Especially if you realise Prevx power is not on-demand scanning, but in realtime (two people were earlier :)). You should take into consideration that there are several problems with MDL, but in my eyes the most important is that 95% of that "malware" (is it really?) doesn't reach any consumer pc's anyway.
     
  11. Triple Helix

    Triple Helix Webroot Product Advisor

    Joined:
    Nov 20, 2004
    Posts:
    12,011
    Location:
    Ontario, Canada
    Also Prevx users with a license have a guarantee of free clean up with a remote session if necessary!

    TH
     
  12. ako

    ako Registered Member

    Joined:
    Nov 16, 2006
    Posts:
    627
    The only point I wanted to make here was the fact (that everybody here should know anyway): the detection rate of 0-day malware of AV:s is too low. In addition to them one definitely should use sandboxing/LUA.
     
  13. Triple Helix

    Triple Helix Webroot Product Advisor

    Joined:
    Nov 20, 2004
    Posts:
    12,011
    Location:
    Ontario, Canada
    For me I would have to say no for the average user! But for ones that like to go find malware then yes! IMO ;)

    TH
     
  14. wearetheborg

    wearetheborg Registered Member

    Joined:
    Nov 14, 2009
    Posts:
    667
    I take back my comments, I did not realise it was 0-day :p
     
  15. ako

    ako Registered Member

    Joined:
    Nov 16, 2006
    Posts:
    627
    LUA? Only for those who "like to go find malware"? Really??
     
  16. Triple Helix

    Triple Helix Webroot Product Advisor

    Joined:
    Nov 20, 2004
    Posts:
    12,011
    Location:
    Ontario, Canada
    I'm sorry but I'm not LUA or UAC user and none in my family either and they never get infected as they are average users and some have NOD32 and Prevx and my Mother Inlaw uses MSE all in admin accounts and there's no problems! Also I'm not a advocate of LUA or UAC but like I said it's IMO!

    TH
     
  17. wearetheborg

    wearetheborg Registered Member

    Joined:
    Nov 14, 2009
    Posts:
    667
    Oh snap :D :D :thumb: :thumb:
     
  18. acr1965

    acr1965 Registered Member

    Joined:
    Oct 12, 2006
    Posts:
    4,954
    Could you break down this math again as I don't understand what you are trying to say. I see where it says HMP detected 1040/1100. Where are you coming up with the 44% assumption?
     
  19. trjam

    trjam Registered Member

    Joined:
    Aug 18, 2006
    Posts:
    9,057
    Location:
    North Carolina
    He is breaking down the number of scanners and the possibility of each finding malware. More is better. I like his test and I also know he is doing a pretty good job.

    He has tested a couple of products for me against the test bed and I would only post if they were positive, so dont ask who they are.

    TH, question about Prevx. If on execution is when really it detects, why does it have a scanner then. I am just trying to understand. I see where people write that on scanning it wont detect, but if something opens it gets it. Why the scannero_O?
     
  20. Sveta MRG

    Sveta MRG Registered Member

    Joined:
    Aug 16, 2009
    Posts:
    207
    Just noticed this thread and want to make a few points.

    With zero day / very early life samples, most AMs will detect between 40 – 60%, depending on age and type of malware.

    A good proportion of zero day / very early life malware never seem to make it in to the wild in any significant way, so testers who use sites like MDL are not using zero day / early life samples.

    A detection rate of 40% on very young samples is fine. We test Prevx regularly on various samples and its as good as the top five.

    One thing to consider is that ALL AMs will miss about half of zero day – very early life samples, however, Prevx will block the action of a good proportion of malware that runs on a system via its SafeOnline technology.

    I know this test was just an on demand test and does not purport to be anything more than this, but, it serves as stimulus. Consider the two main ways security apps are generally tested. 1) Static 2) Dynamic. In our view, dynamic testing, using a realistic vector represents the minimum specification for any meaningful test.

    There will be more and more testing organizations out there which will perform dynamic tests, but you need to look beyond this to really assess the efficacy of applications. For example and with Prevx in mind, you could execute a zero day financial malware sample and this may go undetected by Prevx. Most testers may then conclude Prevx has failed. This is not necessarily so. You need to be able to establish if the sample, once executed has been able to operate and achieve its objective.

    You need to be able to assess if the security application actually prevents the action of executed malware, not just if it stops it executing. Doing this introduces a new level of complexity in to testing and requires the use of custom designed simulators or ethically controlled real botnets etc.

    Given it is a fact that about half of all zero day samples will go undetected, vendors need to start looking at technology like SOL which silently and automatically isolates the browser and other applications from the effect of infection OR some whitelist technology to prevent all unknown code from executing.

    Regards,
    Sveta
     
  21. Triple Helix

    Triple Helix Webroot Product Advisor

    Joined:
    Nov 20, 2004
    Posts:
    12,011
    Location:
    Ontario, Canada
    Because when you open the EXE file then Prevx can read the files during extraction and also try to download other files as most nasties try to do! In my own testing that is what I see as not all files in the EXE are malicious. But Joe or Marco would be a better person to ask such a Question to! ;)

    TH
     
  22. ako

    ako Registered Member

    Joined:
    Nov 16, 2006
    Posts:
    627
    (1-p(detected))^5*1100=1100-1040
     
  23. trjam

    trjam Registered Member

    Joined:
    Aug 18, 2006
    Posts:
    9,057
    Location:
    North Carolina
    yeah, but they are eating dinner.;)
     
  24. ako

    ako Registered Member

    Joined:
    Nov 16, 2006
    Posts:
    627
    It does not hurt. It may add some extra protection. And it makes detection quicker/light. (Moreover the average Joe wants to scan to feel safe.)
     
  25. acr1965

    acr1965 Registered Member

    Joined:
    Oct 12, 2006
    Posts:
    4,954
    Maybe I am missing something as I never claimed to be the brightest in the world. And I've never been accused of being up to date. But is it en vogue to guesstimate detection rates regarding such a small sample size? Shouldn't there be some way to find out how many detections each of these antimalware scanners/engines made instead of guessing? I realize there is a total number of samples detected. But isn't it possible one engine detected a lot more than some other? And some other engine detected a lot less than the average? I think Ikarus usually detects a lot more than Dr. Web and I'm not sure if Emsi is just the antispyware scanning or the Ikarus av and Emsi antispyware. But if there was one engine that usually detected a lot (Engine A) and another engine that did not (Engine B), wouldn't it be more likely that all the detections of B were included in A?

    I think the sample size would skew things somewhat but when it is small like this it would be better to get actual numbers of each scanner's detection- including maybe each of the A&B engines of Gdata.
     
Loading...
Thread Status:
Not open for further replies.