The danger of AV testing sites

Re: The danger of sites like AV Comparatives.

That is a very logical opinion.

But time to raise the roof on fire with regards these tests.

It is accepted within the AV industry that most modern malware has a shelf life.The bad guys repack code so quickly so by the time a signature is created for a series of related files then they have already morphed at source so what is being dished up at 0 hour still evades detections.

All those testing sites show is whether signatures are created retrospectively for a series of files.

So everyone feels mushy that brand X detects 99.5% of 500,000 samples right ?

Well sadly this is the biggest fraud of both the AV tests and the AV industry.

Over 99.5% of all vendor databases contains signatures for files that will never realistically be encountered again. Going to use the logic 99.5% of signatures in a vendors database detect less than 0.5% of currently ITW malicious code.

On the flip those that are unfortunate enough to encounter new repacked malicious code at source(Which is quite a few or would be no AV industry) well that would constitute the less than 0.5% signatures in a database.

So at source(time of first meeting repacked code) do you honestly believe any pure AV has 99.5% detection rates ?

If ANYONE does then they are very much in error and some of the most popular and respected AV testing organizations just perpetuate this myth.

 

The issue is entirely Objective:

Thousands of Qualified Malware Samples tested by Professional AV-Testers
(e.g. AV-Comparatives and AV-Test) in the Best AV-Testing Facilities.

-VS-

A few Hundreds (at most...) of Malware Samples (of Questionable Quality...)
"tested" by Amateurs who Lack Both Expertize & Infrastructure.

Sorry, but the Professionals look, by far, more Real-Word than the Amateurs...

Let's Not forget the following:

In the Real-World Tests of AVC and AV-Test, Malware Samples are Executed; they are Not just Downloaded!
This reflects exactly what happens on Users' PCs!

To make the Long story Short

> AV-Comparatives and AV-Test have a given AV-Testing Methodology and Results.

> However, the Criticizers of AV-Comparatives and AV-Test have Not offered us
a more Reliable Methodology & Results than the ones of AV-Comparatives and AV-Test.

As far as the ones who want to know about the AVC Funding, then, they may have a look HERE

 

Re: The danger of sites like AV Comparatives.

How throwing a huge number of real ITW malware to AV with default settings is by any mean "innacurate" for you ?

If one is willing the results to be checked by third parties, that's why there is other independant testers, such as av-test.org. Anyone is free to compare results from various sources.

About the story you are quoting, never heard of it, will check that.

Regards,
Guillaume.

 

Re: The danger of sites like AV Comparatives.

You have valid arguments.

I have checked that myself at work, when a security consultant show me a malware KAV blocked, he repacked it in another way and it installed blissfuly with no warning from KAV.

That also means that in a targeted attack, detection would be very minimal... far below 90%.

Regards,
Guillaume.

 

Another very important aspect to remember is the following:
Reading the current scores of some AV programmes that have been tested and making a decision based on this alone will quite result in wrong conclusions. Why?

Many who utilise the testing organisation's ratings do not study how the testing is done and for what the testing organisations are looking. Also often times the testing organisations are looking at/for the wrong thing as well. An example would be if some potentially malicious file were downloaded onto a computer but if that file just sits on the hard drive and nothing further happens then that file is innocuous. Only if that file executes would that file be a danger, correct? Correct. Well the testing organisations would for the most part attempt to learn if the AV programme being tested actually "caught" and called out that file as it landed on the hard drive but without executing. So far that file would not be any kind of danger to the computer so the results in most testing would be a "Fail" if a programme being tested did not announce or try to stop said file even though no harm is done up until the point that it executes.

By this situation many who place considerable weight on testing organisation's "results" have come to an erroneous conclusion about the effectiveness of any given AV programme. If an AV programme actually stops a malicious file from executing all the time then would that AV programme not be a very good one with superior performance? Yes it would.

Another problematic situation would be say if some legitimate programme had been downloaded and was being installed and it contained some Trojan malware which dumped three files onto a hard drive but those malware files were not executables (by design). Later, a certain fourth file which might be downloaded in Trojan-style would, if executed, utilise the first three and together cause a complete malware attack such as a DDOS or the like. Which AV programme is going to stop every file some malware utilises even if many of the files have no threat potential in and of themselves? Well some AV programmes may "catch" those three files resulting in some AV test's higher "score" but those files are not in and of themselves harmful. Would the computer user be safer if those non-executable files were "caught"? No. Those three files cannot execute and hence are no threat.

So all in all - it is very needful to know exactly how these testing organisations do their testing and it is essential to know how different types of malware infect and cause real damage to a computer or user. Based on the above examples we can see three things:

1) Some mediocre AV programmes may achieve a higher score due to their "catching" files that are innocuous;
2) Some AV programmes may not achieve a high enough score to reflect their actual competence in stopping malware from executing on one's computer even though the executable may be on the hard drive not executing and it having not been "caught" until the time of execution;
3) How an AV programme works as intended by it's developers is critically important to know when judging tests by AV testing organisations

Thank you.

 

An excellent post and a refreshing read.Sums everything up very nicely and sincere thanks for posting.

 

Agreed, also keep in mind one very important fact. MANY programs score in the 99-100% range. Given that, we should actually 'never' seen an infected machine. Many of us run into infected machines daily, and given the statistics above, that shouldn't happen. Which is why I place very little weight on these tests, and more on real world experience. Real world experience is - if you install a product on a high usage machine. Come back 3-6 months later without it being infected (verified by HMP+MBAM, etc), then find 300 HTTP blocked or quarantined items, the product is pretty fabulous. But how many of us have had these so-called 100% programs installed, only to come back and find rabid infections on the same machines? I certainly have..

Three products I REALLY like, Webroot, Comodo, and Bullguard, will almost always score lower on synthetic tests, but actually preform remarkably well during real usage, far far outstripping the performance of the so-called 100% products when actual usage day to day is factored. I use the above products because I can be pretty assured I will come back to high risk machines a few months later, and find them totally fine. Not many products offer that level of protection.

 

@Q Section
Excellent post, thanks

@Bodhitree

Although I perfectly understand your point, I have a different experience. Every infection I see are from people who either :
- have an outdated antivirus (never updated, not registered)
- have no antivirus
- have AV brands not listed on av-comparative
- have a free AV I won't mention because I'm not here to bash
- have an AV requesting for action and they click "allow"
- never update Windows or their applications

I see no contradiction between scores like 99-100% and real world infections I see every day.

But that is just my personal experience.

Regards,
Guillaume.

 

I don't know, the last box I worked on had 1600 trojans or traces, with Avira installed at the start, and of course auto updated. Given Avira's robust scores, this really shouldn't have happened. Which again is why I place much less stock in these tests, and I think that's a safe thing to do. If I had installed something with a 'quiet' blocker, and strong HTTP filter, I suspect they wouldn't have gotten infected at all. So a 99.8% product vs a 97% one with super strong HTPP scanning and HIPS, I will take the lower scoring one 100% of the time.

Synthetic tests put people at risk if they use them as the sole guide toward what product to get.

 

OVER-Exaggerating yourself?
A box with 1600 Trojans or so could Not start Windows in the first place....

 

@PJC: I don't disagree with you; they are professionals for a reason. But all testing has limitations. That includes everything from school exams to aptitude tests to protection tests like this one.

Just like an exam can not take into account the thinking process of the student who writes it (ultimately, you get the answer or you don't), tests like this are unable to account for working paradigms of every product (good example: Matousec vs. several products, AV-C vs. Webroot).

In history, there are people in for e.g. science who had little formal education, or extreme difficulty studying in that area. That's where the experts failed; because their statistically correct scenario could not accomodate for original thinking. These tests are no different.

So what does this mean? To a layman, this means you can still trust these tests; but your own experience is PRIME. 100% in a test does not mean 100% in real world. You have to consider what scenario you are using it in and what type of product will suit your needs best for the kind of work you are doing. A "standardized" test cannot do this as it is a reference point, not a specific indicator.

That's why for e.g. universities try to match a student's interests with the work at their departments and look at credentials before admitting them for Ph.D. instead of just looking at the test scores (though test scores do matter).

So what I'm saying is what AV-C says effectively: A product that just passes in these tests is good enough for real-world use and beyond that pass point, the differences in detection rate do not matter nearly as much as the suitability of the product to your specific scenario/method of use (though Webroot failed the last on-demand comparative). This is similar to admission scenario: If the aptitude test score is above some specific threshold, then beyond that other credentials matter more than how much higher than the threshold a candidate scores.

So yes, the tests are reliable, but they are a reference point only. It is very important to note this. Thus, a product that scores 90% may actually be better in my experience and for my use than one that scores 98%

 

I think that the issue here is that a good score could just mean that the company is good at the tests, and a bad score could just mean that they're not good with tests. For the former they can make changes to detect the test samples without increasing overall detection in a meaningful way (detects the variant being tested without detecting any of the others, for example), and for the latter they could have a strategy that provides great security overall but doesn't look good in the tests.

Unfortunately there's not really anything else we can go on as far as detection rates; it's silly to think that amateur tests are somehow more reliable, though, as if those with experience and expertise are somehow less knowledgeable than someone that casually reads layman-oriented info that traces back to those same experts. Seeing a few machines with more or less malware is also unreliable, since there are any number of factors that you may not be able to consider.

So we hope that the tests provide some kind of reliable indication. Most importantly, those like us that read up on the issues and stay abreast of tests (over time) and the issues learn how to develop viable security strategies (hopefully), and know that nothing will ever be 100%.

Overall, though, I think that end-users are smarter than a lot of people give credit for. People generally know that product tests (products in general, not just antivirus) are not absolute, they know that antivirus products aren't 100%, and so on; just like any security product (door locks, car alarms, etc.). End users just don't have the interest to spend a lot of time on it, so they look for whatever measure they think they can trust in, and/or they look to people like us for recommendations and guidance. If they're looking at the tests then they're already doing better than most.

 

How 90% can be better than 98% ?

 

Quite easily.

Since you dont know what % of those samples are current ITW malicious code and what % of those detections are malicious code that will never be encountered again.

Detections against malicious code that will never be seen again outside of personal or company repositories are irrelevant.

Detections versus malicious code that is actively distributed today is highly relevant.

 

We used to get more of that kind of information in tests. I don't know why we don't anymore.

 

This holds for the 90%, too; not for the 98%.
So, it is Not that Easy/Safe to infer.

You can Never be sure about that...
BTW, I would Not Easily called the -personal or company repositories- Infections Irrelevant...
There are Still Infections, and therefore, they can be used to Test the Performance of AV products.

 

Very faulted way at looking at modern malware as for most it now has a very short shelf life(In some cases with the new commercial crime packs only hours or X amount of downloads).

All that testing old stuff proves is the ability to detect something after it no longer matters.

Granted it might give some the warm fuzzies if their brand detects 99.5% of samples where as their peers score less but ultimately in the real world usage these figures which get used to sale and promote applications are not the reality when encountering new malicious code,far from it.

The bad guys always get to serve up the wares first, the AV industry can only counter after the fact.

 

Absolutely. Which is why I feel a more valid test for consumers to base product selection on would be a form of real infection testing. If anything, some of those youtube tests with malware domains are probably more valid then these large repository tests. A product can score 99-100% on a repository test, yet utterly fail to protect a system because it doesn't have effective HTTP scanning.

 

Re-read.. 1600 trojans or 'traces'.. Traces qualifies as reg entries, dlls, dead files, directories, etc. Of course a system would run with 1600 traces, that's actually not all that uncommon on an infected machine. My point was, the mentioned product is a 99.5%+ product, and given those statistics it should not have seen a single one of those threats - or traces. Which shows the inherant weakness (at least to me) of the synthetic test protocols. I could install any number of 'strong' products that score lower on synthetic tests and not have had that happen because of effective HIPS or HTTP scanning.

 

(According to your way of looking at Modern Malware), what's the Age of Malware that really matters?

Then, 'why some AVs STILL canNot block old stuff'?
Because, looking Only at Modern malware is a very faulty way...

Since AV-C and AV-Test do Not impress you, then
-What kind of AV-Testing Methodology do you propose?
-Are there any published Testing Results of the AV-Testing Methodology you prefer?

 

In the Real-World Tests of AVC and AV-Test, Malware Samples are Executed; they are Not just Downloaded.
This reflects exactly what happens on Users' PCs!

I've seen many YT tests associated with questionable Malware domains.
"Tests" performed by individuals who lack both Expertise and Infrastructure!
Considering these YT "tests" more valid than the ones of AV-C and AV-Test is not so prudent...

Likewise, a product can have an effective HTTP Scanning, excel these YT "Tests" done by 'Amateurs'
BUT fail to protect a User from even Outdated Malware!

 

That isn't actually totally correct. To conduct a real world test, you need to surf, hit some bad domains, download some junk, get some bad emails, toss a USB drive in, and other things. There are a lot of variables that need to be considered, and tossing a huge stack of largely irrelevant samples at a product, then declaring one great, is doing a bit of a disservice to non-discerning consumers.

Real world is installing a product and getting a conduit toolbar injected all over your system. What is interesting is, these toolbars actually behave like malware in some cases, yet few products block them. I was most impressed to find NOD32-IS blocks virtually all of these under the "PUP" category. This fact alone would prevent a huge amount of headaches on the average users PC. Another factor which AVC neglects to inform, HTTP blocking is absolutely crucial for everyday computers. Some products are astounding HTTP blockers that themselves will get between 90-100% of threats prior to their arrival to the users system. AVC neglects this very important aspect totally. BG licenses the very advanced, and expensive Commtouch enterprise suite which is the most advanced HTTP filter in the world - AVC would neglect this extremely important aspect of the product.

Using the car analogy, AVC would be like starting up the car you want to buy, and standing next to it to determine if you want to buy it. Real testing would be to get in it, play with it, see how it feels, check the features, then take it for test drive. To be honest, a lot of the youtube testers are actually providing more valuable service then ones like AVC.

 

@Bodhitree: please read the reports of AV-C and AV-Test before commenting them. Both AV-C and AV-Test take into account also HTTP blocking etc. and doe exactly the same things as users do in their real-world tests.
In the first post you also stated that BullGuard 2013 was not tested, which is also wrong, as you can read even on the BullGuard website.

 

Had to to search these out as they were informal tests just for referencing but from the details Bruce supplied to me the file(s) were only drawn from the 1 source and the infection vector was an exploit kit on a compromised site.
FYI *Driveby infections* are probaly the most common sources of malware infections currently.

http://forums.malwarebytes.org/index.php?showtopic=112556
http://forums.malwarebytes.org/index.php?showtopic=112749

The general crux of tests show that if you were replying on pure AV protection(static file sniffing not Security suites with multiple tools) then at 0hr then good chance your brand would fail..then go 0 +6, +12,+ 24 hrs a few more arrive at the party but frequently the source repacks/morphs and its back to 0 hour again when someone first encounters the new file.

As i said in my first post in this topic by the time the quicker AV companies release a signature for a series of files, the bad guys already release 1 or 2 revisions that are 0 hour.

If this pattern is repeated across a proportion of the malware spectrum then delayed tests really are not worth snuff(Except for product promotion/sales and keeping Pro AV testers in a job).

Personally i treat all tests with a pinch of salt unless i conduct them myself as to what i would recommend then the only kind of test that holds any validity for pure AV is the 0 hour new code live testing scenario.

I have seen some good amateur takes on this on YouTube but that not to say all you Tube testing is good(Far from it,for the most its extremely bad).