AV-Comparatives Real-World Protection Test - July-August 2022

Take a look at THIS old Wilders thread on Mamutu -- from back in the days of auld lang syne.

 

No thanks. (although did glance and run through some posts there) SIGH. That is almost like teasing. Before we veer too far off topic it still is boggling that there are no pure BB's if nothing else from freelancer open source projects. Yeah i know times change, but Windows certainly hasn't. Can you imagine a ransomware trying to sneak past a quality Behavior Blocker?

Even any of those now once well known relics would jump at it. Getting back to tests, AV-Comparatives-Real World Protection Tests almost always pits the same players and it just doesn't seem it's credibility is what it could and should be since they like us to believe one AV was awarded a better detection than Brand C etc.

But be that as it may, at least it gives them something for others to either complain about or joy in that their Brand X did better this time then Brand Y etc.

 

So, it would appear that Panda is attempting to address its Achilles' heel respective to false positives. If Panda could turn this around into a trend in tandem with their consistently stellar "blocked" performance... Nice!
On a different note, it's at least entertaining to gaze at this graph with all of that red to Malwarebyte's detriment perspectively when the fact of the matter is it was outshined by Bitdefender by a whopping mere 1%.

 

Guys, I think you're misunderstanding me. I'm actually a fan of user controlled behavior blockers, but I'm just saying that it's not the job of an AV. So I'm sure some of them have BB's that require users to make decisions, but to me this is always a fail, since you can't expect the average user to make the right choice.

It's similar to Win Smartscreen, which simply tells you that it doesn't recognize certain apps, because it's probably not on the whitelist, that's not what I call security. So I don't care about how AV's decides that some app is malware or not, this can be done with heuristics/BB/cloud or local signatures, but it should always be a ''yes or no'' answer, not a ''this might be possibly malware.''

 

I had Emsisoft Business installed. Behavior detection blocked several actions that I thought were harmless. In addition, EDR warning messages came up all the time, which I didn't understand at all. Now I'm using F-Secure and I no longer get any warnings or blocks. I suspect the actions of Emsisoft Business would have caused the software to stop working for me. F-Secure is clearly the more suitable AV software for me.

 

In my prior comments, I made 2 points (more than once) that:

!- AVs with aggressive Behavior Blockers are mainly needed by higher-risk users only.

2- AVs with BBs that seldom or never pop up alerts have used methods that will somewhat reduce their scope of protection. (BTW, this is not necessarily a fault. Avs that have few alerts are almost always quite enough for most home users. In fact some users have gotten by for years, free of infection, with no real-time AV at all.}

 

Heuristics are not able to tell if something is malicious or not, only a person can do that. So by your logic, antiviruses shouldn't have heuristics. A heuristic detection indicates that the file could possibly be malicious. The heuristics in most antiviruses is severely limited by the need to reduce false positives. So rather than flagging every file that many be malicious they look for multiple indications that a file may be malicious, in order to greatly reduce the amount of false positives. Although in some antiviruses you can increase the heuristic sensitivity, in order to detect more malware at the expense of more false positives.

With behaviour blockers, it's very hard to for an antivirus to determine if an action is definitely malicious, which is why users are sometimes prompted to choose an action. Sure, users won't always make the right choice and as a result get infected. But sometimes they will and therefore won't get infected. If you remove the prompts, then you remove the possibility of users making the right choice and not getting infected.

Your SmartScreen example is a form of security and will prevent some infections. If you don't want to use SmartScreen you can disable it.

No antivirus, no matter how good it is provides 100% protection. However using the above measures help improve protection. If you were to remove them, then they would provide less protection, leading to more infections and that's definitely a bad thing, even if you don't like them.

 

If you guys want to, or better yet, if the Micro Windows Industry really and truly wanted 100% clearance and proof of some potential malform disguised malware or perfectly safe 'Unknown', i can only see a Virtualization/containment grab of a file which triggered a Behavioral Blocker before or after an AV has given it a pass. It's a shame the lengths security programs like AV's and even others (3rd party) are forced confronted with, but in a contained/restricted state (until said file/process is 100% cleared) what other logical choice remains?

 

I don't know why you're making such a fuzz about my comments. I already explained that in the context of AV testing, I don't believe that ''user dependent'' should even be a thing. I would count this as a fail, same as you also need to take false positives into consideration. So I'm not saying that any detection technology should be removed from AV's, but I'm saying that AV's should always come with a clear verdict, an app is either malicious or not.

 

In a perfect world, having perfect AVs, I would agree with you.

Lacking a perfect world, the current state of AV-tech requires that increasing an AV's aggressiveness will generate more alerts requiring user decisions. Higher-risk users NEED that type of AV. If an aggressive AV is designed for those higher-risk users***, are we to say that its alerts constitute "failure"?? GOOD grief, of course not!!!
~~~~~~~~~~~~~~~~~
***Some not-higher-risk (but paranoid) users also want aggressive security apps.

 

But we all know that any software that requires a users decision to "allow" or "disallow" will only work for a very select few people because a very large majority of people will click allow all the time or have no idea what they are doing.

 

What you are are asking for is an AV that is full-on default/deny with NO possibility of user intervention at time of denial.

A full-on default/deny AV MUST either:
1- Be *perfect* so that it never denies a non-malware app.
OR
2- Use various imperfect techniques to cut FPs/alert to bare minimum.

As to AVs in category #1 -- We don't have the technology to produce perfect default/deny AVs. Moreover, a full-on-but-imperfect default/deny AV would soon come to be regarded as a worse PITA than the malware it was designed to prevent.

As to AVs in category #2 -- Many such AVs exist and offer protection that is quite sufficient for many users. Those AVs have achieved minimal FPs/alerts largely (NOT totally) by reducing their scope of protection. Consequently, those AVs have somewhat increased their vulnerability to 0-day malware, as well as to malware using techniques not yet detectable by existing signatures.

BOTTOM LINE: Higher-risk users need and want AVs that offer a greater scope of security than is offered by Category #2 AVs. The lack of a "perfect AV" makes it unavoidable that aggressive AVs will often produce a number of FPs/alerts requiring user decision (see NOTE 1 below). If doing that job effectively is re-defined as "failure" then I see no reason why aggressive AVs would even participate in reviews that cater solely to users who do not want aggressive AVs.
~~~~~~~~~~~~~~~~~~~~~~~
NOTE 1: I am not saying that the number of FPs/alerts are a sole measure of an AVs degree of protection. Of course, a poorly designed AV of any degree of effectiveness could also produce lots of FPs/alerts.

 

True as summer rain @digmor crusher. That sole fact alone makes advanced AI and/or machine learnings in security programs being actively developed and improved on, ever the more an important worthwhile endeavor as i see it. A certain degree of successful detection and captures can be achieved despite critics and even in spite of detractors simply because if enough math formula's (algorithm's as it's widely called) are finely or granularly applied, and coupled with other magnificent inventive initiatives (innovations), shortcomings for developers of those type programs should make it easier to build or add even better techniques.

Microsoft Windows sure has weaved an incredible spaghetti concoction of paths, entry points and more importantly, have coded language by the zillions intertwined and malware specialists are alert to exploit any they come across. The classic cat & mouse scenario but thank goodness Windows inner workings are finite, many that they be. It's just a matter of auditing them all. No easy task for most skilled in my opinion but yet achievable with the help of some AI to automate the process.

And all that is my take because you are 100% right! Most people wing it and will happy click to their heart's content expecting their AV will pick up the pieces when they open the door to their machine's. Also after all, who has the time or wants to devote so much of their time to constantly 'learning' what's best for their computer security when they can stick an AV on it and let it do what it's intended purpose is supposedly designed for in the first place.

 

If antiviruses would only indicate if something is malicious or not, then they would have to go without heuristics and behaviour blocking. As I've already mentioned, heuristics can't tell if if something is definitely malicious. The same is largely true for behaviour blockers too. Only detections from signatures indicate that something is definitely malicious, aside from occasional false positives.

If you are talking solely about testing, the test results show what amount of detections were user dependant, so you do get a clear indication of what threats were detected and quarantined automatically.

 

I do not see so much as ONE of the previous comments that "criticizes" or "detracts" from AVs having zero or near-zero FPS/alerts. To the contrary, I have repeatedly commented that such AVs are quite adequate for many users who are not higher-risk users.

System Administrators, ITs, and others whose reputation & livelihood are highly dependent on their system's security WILL most certainly "have the time."

Of course this is equally true (probably more so) as relates to System Admins dealing with the security of military and intelligence information.

This is a very plain and obvious truth so I can't help but wonder why it has generated so much debate.

In short, what exactly is the problem with having some AVs employ methods that are designed to attain higher aggressiveness for higher-risk users?

There many more than just a few higher-risk users whose livelihood makes them very very willing to evaluate behaviors with medium to high possibilities of emulation by non-malware.

The typical user of friendly security apps -- those of them who are annoyed by security apps requiring user decisions, or are apprehensive about making such decisions -- they are VERY seldom the main target of malware developed by nations and other highly skilled malware developers.

It is almost always the higher-risk users who are the prime targets of malware developed by nations and other top-grade malware writers. Those users are quite willing to employ security apps that sometimes require them to make security decisions.
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
The latest Air Bus passenger aircraft is over 99% operated by computer AI & other algorithms. But they still have Captains and Co-pilots in the cockpit. Computer operated automobiles and buses still have accidents.

Will the day come when airplanes, automobiles, and security apps need NO human intervention whatsoever? Will the day come when computers can totally take over most repetitive-type jobs, and even mid-management decision-making jobs? Many folks, some of them "experts' in their field, say the answer to those questions is YES! If they are correct, I hope it's not in the lifetimes of me or any of my family.

 

I fully agree with you!

 

AFAIK, it's not true what you say, because heuristics and behavior blockers can also play a role in deciding if some app is malware, what do you think happens in the cloud? What I'm talking about are ''dumb'' behavior blockers who simply alert about suspicious behavior, but leave the end decision up to the user.

But anyway, enough talk, I know you're an AV fan, can you post some screenshots of these kind of alerts? What do they look like, because I have never seen them before. As said before, Win Defender never gives these kind of alerts. I do know that Kaspersky has a full blown HIPS, but I assume this is turned off during malware testing.

Well, this is indeed done in these kind of tests, but in my view it should be counted as a fail. I remember in old tests when Win Defender often performed badly, people said that Win SmartScreen should always be enabled, because then WD would detect more. But this is completely false, since SmartScreen doesn't actually know if some app is malware or not.

 

BTW, take this as an example. Most well known AV's could identify the RedLine data stealer on VirusTotal, and I assume some of them block it with probably heuristics/BB, and others with signatures, either local or cloud based. And I assume they automatically block this stuff, without giving the user an option to allow execution.

https://www.bleepingcomputer.com/ne...cked-help-desk-targeted-players-with-malware/

 

I use 2 security apps for higher-risk users, namely OSArmor and Spyshelter (for its antikeylogger and HIPS). For the first few days there were several alerts, which were easily handled. One such alert told me that Wise Disk Cleaner uses takeown.exe, an app also used by some malware. With a disk cleaner, it's okay but it would NOT be okay if (for instance) a notepad app used it.

After 4-5 days, the alerts became fewer and fewer as OSArmor & SpyShelter "learned" my computer set-up. Now they are boringly quiet. Thus, I have no specimen alerts to show anyone at the moment

My family & friends have had similar experiences -- even when used by "regular folks," these aggressive security apps quiet down. They become like a lionapar -- that's a cross between a lion and a parrot. They are usually very quiet but -- when a lionapar does talk -- you BETTER listen!!!
~~~~~~~~~~~~~~~~~~~~~
BTW, even though Wise cleaner is okay using takeown.exe, I decided to permanently block it from doing so. I am leery of most any non-security app that gropes my computer's knickers. Paranoid, wot?

 

My point is that they can't decide if an app is malicious or not, only if it may be malicious. If they block something, then most likely the AV is fairly confident that there's something malicious happening. But there is a chance the AV may be wrong. Once again, the only indication that something definitely is malicious is a detection from signatures.

If something is detected by name on VirusTotal, it is from signatures. If it's a heuristic detection then of course it's from heuristics. But, since VT only does static scans rather than executing the malware, behaviour blockers are not used.

Here's an example of an alert from Kasperksy.
https://www.kaspersky.com/blog/antivirus-notifications/16900/

 

This is not how I understand it. AV's don't only auto-block malware when malware hapens to be in the signaturelist, nowadays they send it to the cloud where behavioral monitoring is trying to come up with a verdict. That's why you get to see certain detection names like: Mal/Generic-S and TR/Dropper.Gen, they are suspicious enough for AV's to flag them. And thanks for the pic, I suppose this is what they mean with user dependent. Like I said, they give the user to allow it anyway, and thus it should be counted as a fail in my view.

 

I think you're misunderstanding. I was talking about the alerts that AV's might give. I'm using OSArmor and SpyShelter myself. Obviously, these kind of tools are not geared to average Joe's, since they depend on a user's knowledge about what's normal and abnormal app behavior. And that's my whole point, this should be the job of an AV.

 

I agree, but I like a product that provides a choice. If you want it to do everything for you, let it do so and accept when it makes mistakes. If I can toggle a setting that let's me decide, that is my preference. As I have stated before, I haven't had anything malicious successfully get in in the last 15 years. I have had Windows installations made unbootable by false positives a few times in that same time period.

 

Yes, but I believe it's not about this. I also think it's a cool extra feature to have, but in AV testing it shouldn't play a role, that's all I'm saying. AFAIK, Win Defender doesn't give such alerts, so it doesn't have a user controlled BB, for good reason because it might cause too many alerts, and then everyone would complain again.

 

I can agree. The default should be no prompts for testing. But I like options for "advanced users".