Prevx1 detection rates falling down?

Discussion in 'other anti-malware software' started by ako, Mar 23, 2007.

Thread Status:
Not open for further replies.
  1. ako

    ako Registered Member

    Joined:
    Nov 16, 2006
    Posts:
    627
    A very interesting study on early (almost zero-day) detection of malware:
    http://winnow.oitc.com/AntiVirusPerformance.html

    Here the performance of prevx has dropped quickly. From position 6-7 to 13 and detection rate from ca. 45% to 35%.

    And, if you think about it: the result is cumulative, so the performance has actually collapsed. :'(

    Perhaps there is at the moment something wrong with the implementation of Prevx on Virustotal?
     
  2. Old Monk

    Old Monk Registered Member

    Joined:
    Feb 8, 2005
    Posts:
    633
    Location:
    Sheffield, UK
    Hi ako

    I see no 'study' on this link, just a graph o_O This means nothing in itself as far as I'm and many others would be concerned. Got anything else ?

    Oh okay -skip that I see the links at the bottom. I'll let more knowledgable comment further
     
  3. ako

    ako Registered Member

    Joined:
    Nov 16, 2006
    Posts:
    627
    Click the link below the graph. The study is legitimate and good.
     
  4. Old Monk

    Old Monk Registered Member

    Joined:
    Feb 8, 2005
    Posts:
    633
    Location:
    Sheffield, UK
    Hi ako

    Yes, I edited my original post having noticed the links. As I say, with that in mind, I'll bow out and let those here who are knowledgeable in AV comparisons comment further.

    No offence :)
     
  5. lucas1985

    lucas1985 Retired Moderator

    Joined:
    Nov 9, 2006
    Posts:
    4,047
    Location:
    France, May 1968
    Prevx only uses the database in Virustotal. No heuristics, no behaviour analysis.
     
  6. ErikAlbert

    ErikAlbert Registered Member

    Joined:
    Jun 16, 2005
    Posts:
    9,455
    Not a problem for me, my frozen snapshot takes care of what was not detected by my firewall and all the rest of my security softwares, like Prevx1.
    The only reason why I use Prevx1 is because I don't understand the other HIPS softwares.
    I'm also very interested in Faronic's Anti-Executable, because I can use it in my frozen snapshot to save me during the day and AE is based on a WHITELIST, that is based on what is installed on my computer and that method reduces the volume of the whitelist to a minimum on my harddisk.
     
  7. Perman

    Perman Registered Member

    Joined:
    Nov 23, 2005
    Posts:
    2,160
    Hi, folks: What goes down must go up again. So is Prevx1's detection rate. Prevx1 is sold to users mainly by its philosophy and concept(I called it revolutionary, and innovative). No single app can beat Prevx1's many many informants(I mean database), and its mesh size of filter is so fine that few few malwares can have slight chances to evade its detection. It has my full vote of confidence. Besides, like ErikAlbert, I use DeepFreeze, that will strengthen the immune system to the next level. Malware infection is not my primary concern any more; I spend more and more cybertime in surfing, playing games and testing new apps, of course w/o any fears.:)
     
  8. trjam

    trjam Registered Member

    Joined:
    Aug 18, 2006
    Posts:
    9,057
    Location:
    North Carolina
    The funny thing is with Prevx, they are sitting on a gold mine as far as product describtion. It is time to get their latest release out of beta.
     
  9. ako

    ako Registered Member

    Joined:
    Nov 16, 2006
    Posts:
    627
    As you can see by looking my avatar, I'm a Prevx1 fan, and definitely believe in the concept. I'm also active Prevx1R-beta tester. I just find it very curious to see such a sudden drop of detection, and would like to see the explanation.
     
  10. Lucy

    Lucy Registered Member

    Joined:
    Apr 25, 2006
    Posts:
    401
    Location:
    France
    Souldn't it be rather:

    Si vis pacem para bellum

    o_O :D
     
  11. trjam

    trjam Registered Member

    Joined:
    Aug 18, 2006
    Posts:
    9,057
    Location:
    North Carolina
    I really dont think there was a drop. This is more inline with reality.
     
  12. ErikAlbert

    ErikAlbert Registered Member

    Joined:
    Jun 16, 2005
    Posts:
    9,455
    The testbed decides who the winner is.
    In another testbed, Prevx1 might have been the winner.
    Prevx1 was just not lucky in THIS testbed.
     
  13. trjam

    trjam Registered Member

    Joined:
    Aug 18, 2006
    Posts:
    9,057
    Location:
    North Carolina
    Can someone show me a testbed where they won. Dont get me wrong, I love the product, but the time has come to do what is right and make it a contender.
     
  14. Seer

    Seer Registered Member

    Joined:
    Feb 12, 2007
    Posts:
    1,596
    Location:
    Singidunum
    Very interesting thread. It seems that no one here dare to doubt in validity of this test. What exactly makes you guys think this test is legitimate? Where are previous tests by this company? That aside, even if it is legitimate, is Prevx1 an AV? I thought it's a heuristic HIPS, AV-comparatives test Prevx1 as a HIPS, right? IMO Prevx1 rates very well with this suspicious testing, as it is being compared to mightiest AVs available. I have never used Prevx1, and I am not biased here at all. But it is just ridiculous to change an opinion about a software by studying a single test.
     
  15. BlueZannetti

    BlueZannetti Administrator

    Joined:
    Oct 19, 2003
    Posts:
    6,590
    The Seer,

    It could also be a situation where people are simply trying to get a firm handle of what the results may or may not mean.

    It's not a test in the usual sense, it's a running compilation of statistics, but I would tend not to focus on the company/individuals providing a public platform for the data. The data are what they are.

    As for some key points regarding the summary, they do state here that:
    so presumably this is a verified subset of what VirusTotal sees. The tabular numbers (here) reinforce this with the total malware sample population of only 1961 (between Dec. 2, 2006 and Mar. 24, 2007). Therefore, the sample population is small, but is likely of good quality and of circulating malware.

    I do not know how the statistics are gathered with respect to Prevx, so do keep in mind as a current or potential user that Prevx provides hybrid black and white listings. From typical VirusTotal analyses, it does seem as though a hit for Prevx is registered only if it is on their confirmed blacklist, which is what one should expect. However, for a user with Prevx on their machine, with a default installation, any unknown applications will be flagged with an allow/block query. I do not believe this is reflected in the statistics maintained, but this is a real factor for an end-user. Naturally, a user can configure Prevx to automatically either allow or block unknown applications in additonal to the default of query.

    At least in the case of some whitelisting capability, it is unclear whether the statistics provide the entire picture pertinent to an end-user. For a pure blacklist approach, it seems to provide a reasonable snapshot, although the global scope of the result (i.e. the real scope of the testbed relative to actively circulating malware) does remain unknown in detail.

    Blue
     
  16. ako

    ako Registered Member

    Joined:
    Nov 16, 2006
    Posts:
    627
    Thanks, pardon me! :)
     
  17. ggf31416

    ggf31416 Registered Member

    Joined:
    Aug 20, 2006
    Posts:
    314
    Location:
    Uruguay
  18. BlueZannetti

    BlueZannetti Administrator

    Joined:
    Oct 19, 2003
    Posts:
    6,590
    ggf31416,

    Well, if everybody registered that opinion, I guess that relegates me to nobody status :( That wouldn't be the first time that happened.

    In any event, due caution does need to be exercised in the interpretation of the results presented. Unfortunately, test results are often viewed through glasses that presume the performance metrics conveyed are accurate and precise to the numeric digits provided. That's true within the confines of the testbed. However, how that quantitatively translates into in-the-field performance remains an unknown, and unknowable, transformation. That applies to all tests, even ones I pay particular attention to such as av-comparatives.org.

    One aspect of this test that appears good is that it does not rely on flagging by one or more AV products to qualify the testbed. That action necessarily stacks the deck in favor of that product (i.e. coverage is expected to be 100% when measured against that tool). In this case there is the explicit statement that the malware samples have been manually verified as such. Presumably, the harvest protocol assures that the samples may, in some reasonable fashion, be considered actively propagating. Again, a good point.

    A negative is the unknown scope of coverage. In 3.5 months the testbed is 1961 samples large so it's growing, on average, by 18 samples/day. If you look at the KAV signature database size (~260,000 entries) and it's current doubling time (~ 20.5 months), that means they're adding on average about 300 signatures per day. In cases where similar information is available from other vendors, similar numbers (i.e. within a small integer multiple/divisor) are seen. What this means is that the snapshot provided is a rather restricted one. The nature of that restriction is uncharacterized. It may weight specific subtypes highly and neglect others, but the key issue is that this is unknown, so caution should be exercised even in assessing rank ordering.

    These comments aren't unique to this test.

    Blue
     
  19. ggf31416

    ggf31416 Registered Member

    Joined:
    Aug 20, 2006
    Posts:
    314
    Location:
    Uruguay
    I mean almost everybody. I'm sorry if I offended you.

    I agree with all you said.
     
  20. Perman

    Perman Registered Member

    Joined:
    Nov 23, 2005
    Posts:
    2,160
    Hi, folks: Any scientific test results or data colltection is always welcomed by me, it is better than hearsay. However, unless it can produce the kind of level of accuracy as some reputable survey reports, such as within +/- certain%, and 95 out 100 times . I would still treat them as reference and info. Nothing can truly beat the truth produced by real people using real app on a real daily life, that is average users' feedback. The impartiality of those test reports are needed a deep scan. How do we know they are not front line foot lancers of someones ? Just curious. o_O
     
  21. ako

    ako Registered Member

    Joined:
    Nov 16, 2006
    Posts:
    627
    BlueZanetti, very intelligent comments.:thumb:

    I want to add, that 2000 samples would be enough for about 2% uncertainty, IF the sample is representative. (i.e. 35% +- 2%)
     
  22. BlueZannetti

    BlueZannetti Administrator

    Joined:
    Oct 19, 2003
    Posts:
    6,590
    Sorry ggf31416, absolutely no offense taken! My tongue was firmly in cheek, but I see that wasn't obvious in the post :)

    Cheers,

    Blue
     
  23. aigle

    aigle Registered Member

    Joined:
    Dec 14, 2005
    Posts:
    11,047
    Location:
    Saudi Arabia/ Pakistan
    Can anyone please tel me what is this product "webwasher", a sort of filter?
    It is rated on top here and I remember it to be on top in an older Anti-Trojan test as well.
     
  24. lucas1985

    lucas1985 Retired Moderator

    Joined:
    Nov 9, 2006
    Posts:
    4,047
    Location:
    France, May 1968
    WebWasher = Antivir + custom engine.
     
  25. Longboard

    Longboard Registered Member

    Joined:
    Oct 2, 2004
    Posts:
    3,187
    Location:
    Sydney, Australia
    Yes noted that :cautious:

    Comments from Stefan Kurtzahls and IBK as noted.
    fcukdat had a pointed comment.

    This "graphical representation of zeroday threat response/detection" is an interesting little stone in the shoe so to speak.
    I,m not sure anyone is quite sure what to make of it.

    At CC the PrevX support forum posted this as link when PX was high =No2 as a true representation of how well they were doing because of the inherent problems in "testing" PrevX: they can hardly complain now about how they have been pasted in the assessment from the last few days

    @Blue Z: good posts, well thoughtful and illuminating :thumb:
    AS noted it might be great if OITC " Okie Island Trading Company" ??
    Could give month by month, cumulative data: here are some unknowns about this data obviously; but is is an interesting concept.
    It IS cumalative !!
    Anybody with any "rep cred" contacted the OITC yet?

    Regards

    Any body have any pricing info for webwasher? lol.
     
    Last edited: Mar 24, 2007
Loading...
Thread Status:
Not open for further replies.