AV-Comparatives: Real-World Protection Test – May 2018

It is only as good as the malware samples employed. Appears DNS-changer and scareware samples were not employed against Avira. Even if they were, the variants that you were hit with obviously were not.

Also of note by this statement:

You are running a non-default setting configuration.

Bottom line - AV Lab tests are "best guess" rough approximations on how AV software will perform under default settings for the average user. Note the like worded AV-C disclaimer I posted in reply #5. Also of note is "the default setting" rule is no longer an absolute in AV lab testing as noted in AV-C's recent test of Enterprise products. So these reports need to closely examined for like particulars; not just a review of any graphical results chart.

 

Thanks Andreas.

But I still don't understand this about PUA settings:

You wrote:

According to that, PUA detection is on.
That is not the default setting of Eset.
But your own site tells at https://www.av-comparatives.org/real-world-protection-test-methodology/ :

 

exactly, the public doesn't read the methodology, just those shiny bars and numbers then go into fistfights when their beloved AV score "low".
not saying the samples used are not even mentioned.

 

Very well put. Both the age and the particular type of malware are rarely (if ever) stated and sadly no one ever seems to call them on this omission. Further, it always seemed to me that the Pro testing organizations were (are) being overly kind to the products being tested by their concentration on riff-raffy crap like ransomware instead of more insidious malware such as scriptors which are both harder to detect and more prevalent.

 

If serious 0-days and scriptors were tested they all will score like what? 20% (and i'm kind),it would bad advertisement for vendors (and they wont participate anymore), so it will never happen.
Test labs are AV just hidden marketing teams for vendors, and they all laugh when they see fanboys fighting over results.

 

I would just love to see Ophelia run the tests for just 1 month. There would be much Moaning and Gnashing of Teeth...

 

If you don't mind may I know who Ophelia is,you frequently mention her.

 

It's her (obviously very talented) cat.

 

One also has to perform "due diligence" when employing AV lab tests as evaluation criteria and put in some effort to seek out as much data as possible. Not all AV lab testing is the same. For example, SE Labs last consumer AV product Realtime quarterly test: https://selabs.uk/en/reports/consumers included 75% URL based malware samples and 25% targeted attack malware samples.

Some AV labs also performed AV product testing against specific malware categories. Malware Research Group that I previously reference is known for its quarterly banking and payment protection tests where financial malware samples are used. AV Lab - Poland did AV product testing against fileless based malware last fall: https://avlab.pl/sites/default/files/68files/Malware_Fileless_Protection_Test_EN.pdf and just recently a specialized comparative using ransomware, coinminer, and "bashware" samples: https://avlab.pl/en/best-antivirus-software-2018-based-three-security-tests .

The main point in these tests and any malware testing for that matter is the test scope is of a limited malware nature. Both AV-Test and AV-C perform "full spectrum" dynamic tests that are overall a better evaluation tool to a given AV product effectiveness; again test malware scope is limited but broader than the realtime tests.

 

An "illuminating" read about AV lab testing is given in this 2016 Virus Bulletin conference paper:

ANTI-MALWARE TESTING UNDERCOVER
https://www.virusbulletin.com/uploads/pdf/magazine/2016/VB2016-Corrons-Zwienenberg.pdf

Unfortunately, things have gotten worse since this paper was written as evidenced by the "shenanigans" last year to "accommodate" Next Gen vendors.