I applaud Prevx’s openness to sharing information

To galileo and EASTER: The points you make are exactly what we're trying to convey. I personally haven't been infected in 10+ years, and until I started working for Prevx, I never used any AV/security/etc.

Now I use Prevx 3.0 and still haven't gotten infected, but there are hundreds of millions of users that have been infected, many of which were using security software.

In the end, every decision about security relies on a number of factors - one of which is detection, however, if you had an AV that guaranteed 100% detection but added 3 hours to every time you tried to open a program.... would you use it?

 

I agree - while we don't provide this data to any authority on demand for customer privacy reasons, if required by subpoena or other means, we can easily provide full transaction logs proving that we are not mining confidential information, etc. and that all data is stored securely. We actually have a large number of governmental offices and organizations with security as a high concern as customers and have been privately asked to justify our data, and no objections were found

 

It is indeed not an insurmountable obstacle, rather a speed bump which will hopefully be resolved soon with AMTSO. Reliable testing of real infections is expensive (as I've mentioned earlier). When we do our comparisons internally, it generally takes 5-6 of our researchers a full week to run wide comparisons, and even then the sample sets are still very small.

Currently, most antivirus tests can be completely automated and easily run across a wide group of samples, but as more solutions move "into the cloud", you will start to see more and more vendors drop out of these tests (as many have already).

I'm not sure, and I'm also unsure on what the low/medium/high risk variables are based so I'll have to do some asking and get back to you on this

I have forwarded your suggestions on and will ensure that they are seriously considered.

For what its worth, there will be some very interesting announcements coming soon about the effectiveness of our products. We haven't been put up against a plain detection comparison yet (for reasons previously mentioned in extended verbosity ) but as soon as these reports are allowed to be released publicly, it should shed some light on why our solutions are effective and clarify what sets them apart.

 

PrevX Help, might I mention that AV-Comparatives has been testing McAfee with their in-the-cloud solution Artemis for a while now.

I do think they would be able to test Prevx 3.0 separately or in comparison, quite comfortably.

 

PrevxHelp, doesn’t this same observation apply to the “threats that your current security products missed” statistics now reported on the Prevx website? If perceived bias is the criterion for the release of information, shouldn't the website statistics also be suppressed?

PrevxHelp, I am pleased that Prevx is actively seeking to solve the issue of comparing the performance of its products to the competition. While I appreciate the consideration of costs, this activity should not be viewed as an “expense” but as an “investment” – especially for the company’s marketing and sales teams, in my opinion.

PrevxHelp, it is exactly these types of assertions that need empirical support from independent sources. Your statement may indeed be completely accurate, but in the absence of supporting analyses, some users may be inclinded to view such a comment as “marketing pabulum.”

PrevxHelp, how do you reconcile this statement with your company’s description of these statistics as “a real world measure of comparative antivirus protection” (see post #23)?

PrevxHelp, from your own perspective, what are the top five characteristics of Prevx that distinguish it from the competition and which contribute to delivering a superior level of anti-virus protection for users? (This is your opportunity to post an ‘advertisement’ here! )

PrevxHelp, your willingness to maintain an open mind in the exploration of these issues is commendable.

Vijayind, thank you sharing this very germane insight. More information is available from: Anti-Virus Comparative – McAfee Artemis.

 

Symantec is also working toward incorporating cloud based approaches into its anti-virus products – see It's All About Reputation. I could be wrong, but I suspect that Symantec will not be shy about working with anti-virus organizations to ensure that its new products are properly assessed in comparative reviews.

 

Re: I applaud Prevx’s openness to sharing information

Then I guess Norton Insight was only the beginning - even if that was only for speed. Maybe they'll build upon it...

 

...And Trend Micro joins the trend with its own client-cloud architecture: see Reduce security risks and costs through immediate protection.

(The cloud approach used by Prevx is soon to have substantial competition, it appears.)

 

I am totally confident in Prevx and their confidentiality.

Things quickly change, dont they. My question is, this technology is about "In the cloud" based on detection being seen by their servers and from all their users.

So, lets say that OA gives a FP when someone installs 3.0, does that mean that no other Prevx user has used OA.

 

I have a question about the behaviour exhibited by malware , and I'd hope that Prevx might have some idea of the stats behind it.

I'm wondering what %, of browser exploits, execute themselves in the browser cache folder, or in the system temp folders.

I think myself from reading up on it the % would be quite high.

Thanks
J

 

More information on Symantec’s developing cloud-based approach to security…

Source: Symantec Expands Reputation-Based Technologies

 

The percentage is indeed quite high - I'm not sure of the exact numbers, but the days of exploits executing completely from memory are ending and they do tend to drop the files into the cache/temp folders. The reason for this is that the browser already has access to these folders so they are generally allowed through.

 

The marketing blurbs may make it seem like we have competition but they indeed have quite a way to go until they have an architecture like ours All of the in-the-cloud vendors are currently using it as just a supplement to signatures rather than an actual analysis platform.

 

We are releasing the homepage statistics as informational, not as a comparison test between each vendor, so the bias wouldn't be there.

One of the problems with current AV testing is that many of the tests don't require any effort to actually pass - you only pay the $10-50k and you receive a good rating as they send all of the samples you miss. Sure, we could pay the fee, but why

The charts are giving a comparison between the AVs and the threats themselves - it is there to show that every AV misses threats, not to say that X av is better than Y av.

Off the top of my head:
1) Our community view on threats allow us to find targeted malware, polymorphic malware, and quickly spreading threats far better than the competition.

2) We are the only AV which can work at an incremental level with other AVs so you can use us alongside the other products without negative interaction.

3) Our advanced automated research is scalable so rather than being pressed for resources like the other vendors as threats begin to spread faster, we actually prefer a higher volume of threats as it lets us tune our rules better to detect more automatically.

4) Prevx 3.0 is very small and lightweight, introducing a very low system overhead and keeping everything easy to use for the average user by staying as silent as possible.

5) We have a very versatile development mindset - we are able to add new features quickly and keep up with emerging threats (for instance, the newest MBR rootkit) because of our highly-extensible backend framework.

 

PrevxHelp, shouldn’t two approaches—a signature plus a cloud-client approach—be superior to a one-dimensional, cloud-client only approach? If not, why not?

PrevxHelp, it isn’t about “passing” a test—the key insight is the relative performance of one anti-virus solution to another, as assessed independently in an empirical manner.

PrevxHelp, it sounds as if you are suggesting that anti-virus comparatives are a “charade” —perhaps even a “fraud.” Can you kindly elaborate?

PrevxHelp, while I trust your statement reflects your own perspective, it does not appear to reflect the viewpoint of Mr. Mel Morris, the CEO of Prevx. Mr. Morris said on June 28, 2008: “Daily analysis of security breach data highlights strengths and weaknesses of top brand PC security products – a real World measure of comparative antivirus protection. … The charts highlight some very interesting and surprising strengths and weaknesses between various brands.” Those statements by Mr. Morris certainly sound as if the statistics on the home page of your website are designed to facilitate a comparison between anti-virus solutions.

If you’re right and Mr. Morris is wrong, then it would be worthwhile to clarify the situation by explicitly stating in the “explain this chart” section of your home webpage that (1) the statistics do not allow a reader to make an informed comparison between products; and that (2) Prevx also misses threats that the competition does not, to an extent that may be less, the same or more than other products. Do you agree with this suggestion?

PrevxHelp, what evidence can you share to support the assertion that Prevx finds threats “far better” than the competition?

 

I don't see how a signature approach could ever be better than the cloud approach. Standard AV signatures require the following steps:

1) An AV company receives a sample
2) An AV company analyzes the sample and determines that it is malicious
3) The researcher manually creates a signature for the sample
4) The signature is tested to ensure that it doesn't cause any obvious FPs
5) The signature is uploaded to the AV's database
6) The client software checks for updates (once per hour or so)
7) The update is downloaded and applied
8 ) The client rescans their PC and then the new threat is identified

In our model, the process goes:
1) The sample is automatically blocked or our automated analysis blocks it - everyone is protected
2) We sit back and work on preventing threats rather than keeping up with existing ones

Our signatures do some of the same things that conventional AVs do, like black listing, string searching, whitelisting, heuristic analysis, etc. but it does them all in the cloud rather than on the local PC, reducing resource needs and preventing all of the steps required for manual analysis and uploading/downloading.

The only benefit I see for using the conventional approach is that it is less expensive for an AV company to set up but they are paying far more to their large staff of researchers.

The plain fact is that NO av is effective right now. Users are being infected left and right, and infections are growing at an exponential rate. Take Conficker - how come 10+ million users were able to be infected with it? Because the conventional AV model fails miserably. If an AV test says that X av detects/blocks 99% of threats, we simply don't want to be a part of it because it is a lie.

Many tests can indeed be categorized as a charade. I won't name names here but simple research can be done to find that the testing methodology of many "professional" AV tests includes sending vendors the samples which they miss and then offering them the ability to retake the test.

I agree with what our CEO says - it compares the protection of each AV against the real world threats we see every day. Many AVs are stronger than others in different aspects, but we don't go along and rank them - if we were to give the statistics about the ranking, then it would be obvious that we were looking to compare them against each other. A better word than "between" may be "across": the weaknesses span across all vendors, as seen by the charts.

Conventional AVs rely on the steps I've outlined above - we are able to see a threat as soon as it is seen by the first user. Conventional AVs have to look at threats with an economical view where they can't focus on threats that only affect a small number of users. Our architecture allows us to detect threats on either the first or second sighting and there are MANY threats which are only seen by a small number of users today because malware authors are using a targeted approach to be more effective with their home-grown exploits.

 

@PrevX help

I have a question about the free version. I have played a little with PrevX,

Heuristics: only provide option to remove (greyed out, update or trust)

Age: just a way of restricting the focus of PrevX, thought behind it, the newer the program, the more chance it is malware

Population: when (at program install) a blacklisted program is found, I am able to stop it.

Is the above correct (I have set heuristics after age/population)?

Regards Kees

 

PrevxHelp, are you sure? For example, consider that Symantec created 1,656,227 new malicious code signatures in 2008. Clearly, it cannot be a manual process - correct? That would imply that the “researchers” manually create about 3 signatures per second on average, based upon a 365 x 24 x 7 staffing model.

PrevxHelp, it appears that you may have misinterpreted my question. I am not asking if a signature based anti-virus approach is better or worse than a cloud-client or “herd intelligence” approach. Instead, I am asking why you believe that a solution based upon signatures plus “herd intelligence” would be less effective than a solution based only upon “herd intelligence”?

It seems that your anti-virus competition will shortly be leveraging the “herd intelligence” capabilities now offered by Prevx in addition to their wealth of capabilities based upon signatures, heuristics, white lists and black lists. Common sense would suggest that a breadth of approaches applied to the problem would yield superior results – correct?

Also, since some anti-virus competitors are substantially larger than Prevx, the scope of their “herd intelligence” should, in theory, be superior – since they have greater visibility into the threat landscape through their significantly larger user base. As a point of comparison, how many active users of Prevx exist?

PrevxHelp, those are strong words. Specifically, why do you view such an empirical result as a “lie”? Using the label “lie” implies that the anti-virus test is an intentional falsehood.

PrevxHelp, your statement is accurate – but, in my opinion, not forthright. The home page of your website displays a side-by-side bar chart for each anti-virus competitor. The organization and the display of the information is, in my opinion, clearly designed to promote a comparison and, in that way, is misleading.

You are also avoiding commenting upon my suggestion to clarify the situation by explicitly stating in the “explain this chart” section of your home webpage that (1) the statistics do not allow a reader to make an informed comparison between products; and that (2) Prevx also misses threats that the competition does not, to an extent that may be less, the same or more than other products. Based on your prior comments, it’s clear that that you agree with these statements — so, why are they not prominently displayed next to the statistics that you sharing? If your intent is to educate the public, then shouldn’t the public be explicitly provided these warnings about the interpretation of the data? (Parenthetically, the Prevx brand would be enhanced - not diminished - by taking the "high road" in this regard.)

PrevxHelp, my question was “what evidence can you share to support the assertion that Prevx finds threats ‘far better’ than the competition?” Since you didn’t provide any evidence, is it reasonable to assume that none exists?

If your intent is to use the “missed threats” statistics from the home page of your website to argue that Prevx finds threats “far better” than the competition, then how can you do so and maintain intellectual rigor in the absence of knowing the degree to which the competition detects threats that Prevx misses?

The Prevx architecture may, in some cases, be superior to that of the competition. However, "architecture" isn't the same as "results," and users (in my opinion) care about results - i.e., the extent to which their systems are protected. This is why Prevx should take a leadership position in constructing fair and balanced anti-virus comparatives, so that results can be known and can be openly debated. It seems to me that your choices are to either (1) complain about the existing state of anti-virus comparatives or (2) do something about it. A starting point would be joining the Anti-Malware Testing Standards Organization....

 

Symantec has some automated systems, however, the bulk of the standard antivirus model centralizes around manual signatures. The real problem is the fact that regardless of who/what creates the signatures, they have to be uploaded centrally and then downloaded by the end user - a process completely inverted to how our model works.

I think it would be equally effective as a wholly-centralized approach - there is no benefit to holding the signatures/heuristics/whitelists/blacklists locally so I don't see the need to have both in place at all.

The current companies with herd intelligence do have larger user bases than us (we have around 5 million users) but they aren't leveraging the data at all. Most of them use only point signatures - MD5/SHA1 - with absolutely no possibility for heuristic analysis or further interpretation. However, I suspect that in the coming months you will see an abrupt surge of Prevx users which will feed the database which will find more malware, causing the cycle to feed itself and improve automatically.

The intentional falsehood is the false sense of security - antivirus products do not detect 99% of threats that users are experiencing and they do not detect 100% of "in the wild" threats. Sure, they detect 99% of the test sample base, but I can make 1,000,000 copies of EICAR or 1,000,000 infections of a simple file infector and say that every AV finds 100% of threats. Further information on WHAT threats are missed is not provided, and oddly enough, there are vendors pushing for AMTSO to not disclose information on what threats are missed to prevent "helping the malware authors".

The text which we show in the Explain this chart area includes: "You should expect to see a higher number against the more popular security vendors because we see more of these users and consequently a higher number of malware infections." That clarifies that these statistics are based on raw data, not an interpretation, so users are free to (mis)interpret the statistics on their own. As I've said before, we aren't trying to compare the products, just the products to the threats themselves.

I honestly don't know what the other vendors detect that we miss, but I suspect it is very low because we rarely have users coming complaining to us that we're missing threats and quite the opposite occurs when users find us after having been infected with their current AV. A vast majority of our users aren't the fraction of a percent of people that post on forums so you rarely hear the positive stories, but for now, until other vendors have the reverse comparison charts, we're basing our critique of our own effectiveness on our user responses. We guarantee full removal of any malware - that's a big statement to make if we didn't trust our detection and cleanup and the fact that we have been offering this for almost two years now proves that our model and solution is effective, otherwise we would have either taken it down or not fulfilled the guarantee and users would be complaining everywhere.

To comment on your points: 1) we are complaining about it, 2) we've considered it but currently are not joining. Why? Because the need for change has to come from users - not from other antivirus companies, and the users have been complaining adamantly. While AMTSO may indeed create some good methodology for testing in the end, the real test comes from the effectiveness in a real world scenario. The computer security industry is a large industry with tens of thousands of employees and billions of dollars in revenue. In my opinion, it is unfathomably biased to have antivirus companies designing the rules for testing - try to apply this to a field like architecture: the architecture firms would all build a code of restrictions, metrics, and safety measures that would cut costs and reduce overhead. This would clearly be a risk to public safety, and that's why the government imposes building codes - if not, all of the architecture firms would most definitely look for the easiest way to get the job done. Sure, it would be the result of an collaborative-style organization, but would it be fair for the end user/homeowner and would it change anything? The antivirus software of today is far more complex than the antivirus software of yesterday because today's threats are more complex and I believe most, if not all, vendors agree that detection is much more accurate on-access/in realtime than on-demand. Do we really need an organization to realize that the best way to test the effectiveness of security software is to try and infect a real world system?

 

Re: I applaud Prevx’s openness to sharing information

Did you see the brief "Security in the Cloud" discussion at http://www.pcmag.com/article2/0,2817,2333445,00.asp today? Interesting that two of the companies are just using checksums, Norton seems to use more but If you you are not willing to have users be remote collection and evaluation nodes for malware, probably limited in spite of the potential size of your sample space. See also http://rationalsecurity.typepad.com/blog/2007/12/thinning-the-he.html for some interesting thoughts on the subject. Comments from Prevx?

 

PrevxHelp, I think you have made your point: Prevx does not have any interest in ensuring that the “missed threats” statistics on the home webpage are not misinterpreted by users, and therefore is complicit in the confusion. I am disappointed, and hope that over time your position on this issue will be modified.

PrevxHelp, what percent of threats that users are experiencing are detected by Prevx? What percent of “in the wild" threats are detected by Prevx?

P.S.: PrevxHelp, did you notice that the article “The Best Security Suites for 2009: Security in the Cloud” (see post #45) doesn’t even consider Prevx? Why the omission? Prevx isn't in the consideration set for "security in the cloud" solutions, unfortunately.

 

There is no definitive answer to this as there is no measure of the volume of threats globally. I know we are generally one of the first companies to see wide spreading threats and targeted threats because of how our reporting works so we are at an advantage but the best way to assess the effectiveness of an AV is to try it on real infections. We test hundreds of drive-by websites on a daily basis consistently with extremely positive results.

We weren't included in the list being that we aren't a security suite... so it would be incorrect to include us.

 

OT posts suitable for another thread were moved to a thread of their own.

moved here---> PrevX: "the good, the bad and the ugly"

Bubba